Documentation - API Reference - Changelog - Bug reports - Discord
⚠️ Cortex is currently in Development: Expect breaking changes and bugs!
Cortex is an openAI-compatible local AI server that developers can use to build LLM apps. It is packaged with a Docker-inspired command-line interface and a Typescript client library. It can be used as a standalone server, or imported as a library.
Cortex currently supports two inference engines:
- Llama.cpp
- TensorRT-LLM
Read more about Cortex at https://jan.ai/cortex
Cortex:
Before installation, ensure that you have installed the following:
- Node.js: version 18 and above is required to run the installation.
- NPM: Needed to manage packages.
- CPU Instruction Sets: Available for download from the Cortex GitHub Releases page.
💡 The CPU instruction sets are not required for the initial installation of Cortex. This dependency will be automatically installed during the Cortex initialization if they are not already on your system.
Ensure that your system meets the following requirements to run Cortex:
-
OS:
- MacOSX 13.6 or higher.
- Windows 10 or higher.
- Ubuntu 12.04 and later.
-
RAM (CPU Mode):
- 8GB for running up to 3B models.
- 16GB for running up to 7B models.
- 32GB for running up to 13B models.
-
VRAM (GPU Mode):
- 6GB can load the 3B model (int4) with
ngl
at 120 ~ full speed on CPU/ GPU. - 8GB can load the 7B model (int4) with
ngl
at 120 ~ full speed on CPU/ GPU. - 12GB can load the 13B model (int4) with
ngl
at 120 ~ full speed on CPU/ GPU.
- 6GB can load the 3B model (int4) with
-
Disk: At least 10GB for app and model download.
To install Cortex CLI, follow the steps below:
- Install the Cortex NPM package globally:
npm i -g @janhq/cortex
Cortex automatically detects your CPU and GPU, downloading the appropriate CPU instruction sets and required dependencies to optimize GPU performance.
- Download a GGUF model from Hugging Face:
# Pull a model most compatible with your hardware
cortex pull llama3
# Pull a specific variant with `repo_name:branch`
cortex pull llama3:7b
# Pull a model with the HuggingFace `model_id`
cortex pull microsoft/Phi-3-mini-4k-instruct-gguf
- Load the model:
cortex models start llama3:7b
- Start chatting with the model:
cortex chat tell me a joke
To run Cortex as an API server:
cortex serve
# Output
# Started server at http://localhost:1337
# Swagger UI available at http://localhost:1337/api
You can now access the Cortex API server at http://localhost:1337
,
and the Swagger UI at http://localhost:1337/api
.
To install Cortex from the source, follow the steps below:
- Clone the Cortex repository here.
- Navigate to the
cortex-js
folder. - Open the terminal and run the following command to build the Cortex project:
npx nest build
- Make the
command.js
executable:
chmod +x '[path-to]/cortex/cortex-js/dist/src/command.js'
- Link the package globally:
npm link
The following CLI commands are currently available:
⚠️ Cortex is currently in Development: More commands will be added soon!
serve Providing API endpoint for Cortex backend
chat Send a chat request to a model
init|setup Init settings and download cortex's dependencies
ps Show running models and their status
kill Kill running cortex processes
pull|download Download a model. Working with HuggingFace model id.
run [options] EXPERIMENTAL: Shortcut to start a model and chat
models Subcommands for managing models
models list List all available models.
models pull Download a specified model.
models remove Delete a specified model.
models get Retrieve the configuration of a specified model.
models start Start a specified model.
models stop Stop a specified model.
models update Update the configuration of a specified model.
Run the following command to uninstall Cortex globally on your machine:
# Uninstall globally using NPM
npm uninstall -g @janhq/cortex
- For support, please file a GitHub ticket.
- For questions, join our Discord here.
- For long-form inquiries, please email hello@jan.ai.