Check out TurnkeyML-LLM

[jeremyfowers]

Hi OGA community, I would like to let you know about our new work TurnkeyML-LLM, which can do some cool things on top of OGA.

Some features you may find interesting:

• 🚀 No-code command line interface (CLI) and low-code API
• 🎯 Run MMLU and Perplexity accuracy tests on OGA models
• 🌐 Launch an OGA model in a WebSocket server
• 💡 Access OGA models through huggingface-transformers-like APIs (e.g., from_pretrained())
• 🛠️ Suite of tools and APIs is cross-compatible between:
  • OGA models and huggingface transformers LLMs (e.g., use the same MMLU tester to compare OGA Phi-3 vs. PyTorch Phi-3)
  • CPU, GPU, and NPU (e.g., easily launch the WebSocket server on whatever hardware your user has available)

You can learn more about the project in the LLM readme, including the commands to try each of the features above.

The team would love to hear your feedback! You can contact us at [email protected] or by filing an issue on the TurnkeyML repo.

[natke]

This is so cool @jeremyfowers! We will have a play with it and be in touch!