/openrouter-runner

Inference engine powering open source models on OpenRouter

Primary LanguagePythonMIT LicenseMIT

OpenRouter Runner

OpenRouter Runner is a monolith inference engine, built with Modal, used for lots of the open source models hosted in a fallback capacity on openrouter.ai.

Engines

  • vLLM
  • HF Transformers

Getting Started

  1. cd modal
  2. Select a modal app, like the runner
  3. Follow the steps in the project README.

Contributions

Interested in contributing? Please read our contributing guide and follow our code of conduct.

License

MIT