This project uses Auto-GPT to experiment the posibility of using local LLM. Reference gpt-llmam.cpp to build custom API.
To run the auto-local-gpt:
- Put the models you'd like to try to certain directory.
For example:
wget https://huggingface.co/eachadea/ggml-vicuna-13b-1.1/resolve/main/ggml-vicuna-13b-1.1-q4_1.bin
- Set the environment variables to .env
EMBED_DIM=5120
OPENAI_API_BASE_URL=localhost:8000/v1
OPENAI_API_KEY=<if use custom url replace it with model path>
- Run the docker command, which will automatically start the API endpoint at port 8000
docker run -it -d -v <your models directory>:/llama.cpp/models -p 8000:8000 buckylee/auto-local-gpt:latest
- Run Auto-gpt For Linux:
./run.sh