/pi-genai-stack

Run 🦙 @ollama and 🐬 TinyDolphin, 🦙 TinyLlama and other small LLMs on a Raspberry Pi 5 with @Docker #Compose

Primary LanguageHTML

pi-genai-stack

  • The main objective is to run 🦙 Ollama and 🐬 TinyDolphin on a Raspberry Pi 5 with 🐳 Docker Compose.
  • The stack provides development environments to experiment with Ollama and 🦜🔗 Lanchain without installing anything:
    • Python dev environment (available)
    • JavaScript dev environment (available)

How to install the stack on the Pi

  • The compose file use the include feature, so you need at least the 2.21.0 version
git clone https://github.com/bots-garden/pi-genai-stack.git
cd pi-genai-stack

Models (LLM)

At start, Pi GenAI Stack will download 7 models:

Then all the samples of the demos will use one of these models.

Run the samples of the python-demo

start the stack with the demo

docker compose --profile demo up

Use the python with the interactive mode:

docker exec --workdir /python-demo -it python-demo /bin/bash

Run the python files:

python3 1-give-me-a-dockerfile.py
# or
python3 2-tell-me-more-about-docker-and-wasm.py

Use the Python dev environment

start the stack with the Python dev environment

docker compose --profile python up

The Python dev environment is a Web IDE based on Coder Server with the Python runtime and tools. The environment is starded as a Docker Compose service and you can open the IDE with this URL: http://localhost:3000.

python dev environment

To use it remotely (to connect to the WebIDE running on the Pi from your workstation), use the DNS name of your Pi or its IP address. For example http://hal.local:3000 (where hal.local is the DNS name of my Pi).

Use the JavaScript dev environment

start the stack with the JavaScript dev environment

docker compose --profile javascript up

The JavaScript dev environment is a Web IDE based on Coder Server with the Node.js runtime and tools. The environment is starded as a Docker Compose service and you can open the IDE with this URL: http://localhost:3001.

To use it remotely (to connect to the WebIDE running on the Pi from your workstation), use the DNS name of your Pi or its IP address. For example http://hal.local:3001 (where hal.local is the DNS name of my Pi).

Test your stack remotely

First test with a simple curl request

  • The answer time is long because we do not use streaming
  • Where hal.local is the DNS name of my Pi
curl http://hal.local:11434/api/generate -d '{
  "model": "tinydolphin",
  "prompt": "Explain simply what is WebAssembly",
  "stream": false
}'

Or try this one:

curl http://hal.local:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
    "model": "tinydolphin",
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "Hello!"
        }
    ]
}'

Get the list of the models

curl http://hal.local:11434/api/tags

Load another model

curl http://hal.local:11434/api/pull -d '{
  "name": "phi"
}'

✋ llama2 is too big for a Pi

Delete a model

curl -X DELETE http://hal.local:11434/api/delete -d '{
  "name": "llama2"
}'

Blog posts