This repo contains the Containerfile and manifest for an UBI-based image of ollama.ai and the corresponding Kubernetes manifests to run it
The image is published at quay.io/wcaban/ollama:latest
Prerequisites: Must have the NVIDIA OpenShift Operator installed and operational
- Update and apply each one of the manifests to map to your environment
manifests/
├── 01-ollama-pvc.yaml
├── 02-ollama-serve-deployment.yaml
├── 03-ollama-svc.yaml
└── 04-ollama-route.yaml- Once the Pod is running, login into the Pod to pull desired LLM models
# get name of pod
oc get pods
NAME READY STATUS RESTARTS AGE
ollama-serve-6f97989fd5-tfmph 1/1 Running 0 3d5h
# connect to pod
oc exec -ti ollama-serve-6f97989fd5-tfmph -- bash
bash-5.1$ ollama pull mistral:latest
pulling manifest
pulling 6ae280299950... 100% |███████████████████████████████████████████████████████████████████████████████████████| (4.1/4.1 GB, 77 TB/s)
pulling fede2d8d6c1f... 100% |█████████████████████████████████████████████████████████████████████████████████████████| (29/29 B, 1.3 MB/s)
pulling b96850d2e482... 100% |████████████████████████████████████████████████████████████████████████████████████████| (307/307 B, 16 MB/s)
verifying sha256 digest
writing manifest
removing any unused layers
success
bash-5.1$ ollama pull falcon:7b-instruct
pulling manifest
pulling 305c4103a989... 100% |██████████████████████████████████████████████████████████████| (4.2/4.2 GB, 112 MB/s)
pulling 4dec76bb1a47... 100% |████████████████████████████████████████████████████████████████████| (45/45 B, 28 B/s)
pulling 0644cce03f93... 100% |████████████████████████████████████████████████████████████████████| (31/31 B, 25 B/s)
pulling 9ce7398869ad... 100% |█████████████████████████████████████████████████████████████████| (382/382 B, 120 B/s)
verifying sha256 digest
writing manifest
removing any unused layers
success
# once the models download you can list them like
bash-5.1$ ollama list
NAME ID SIZE MODIFIED
falcon:7b-instruct 4280f7257e73 4.2 GB 5 hours ago
llama2:13b b3f03629d9a6 7.4 GB 2 days ago
llama2:latest fe938a131f40 3.8 GB 2 days ago
mistral:7b-instruct-q5_K_M 1041946669e8 5.1 GB 2 days ago
mistral:latest 8aa307f73b26 4.1 GB About a minute ago
openhermes2-mistral:latest e8aa40c72cb0 4.1 GB 2 days ago
# to test a model from within the container
bash-5.1$ ollama run mistral:latest "What is OpenShift"
OpenShift is an open-source container application platform that allows developers to easily build, deploy and manage applications in
containers. It is based on the popular Kubernetes platform and provides a user-friendly interface for managing containerized applications at
scale. OpenShift includes features such as automatic scaling, self-healing, and built-in security, making it an ideal platform for enterprise
applications. It also supports a wide range of programming languages and frameworks, making it accessible to developers of all backgrounds.- The available models will be accesible over the
ollamaAPI which defaults to http://localhost:11434- For deployments to OpenShift using the manifests in this repo get the URL for your application by executing
oc get route ollama-route -n ollama
- For deployments to OpenShift using the manifests in this repo get the URL for your application by executing
# retrieving the route
oc get route ollama-route -n ollama
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
ollama-route ollama.apps.openshift.lab ollama-svc 11434 None
# testing the route
curl -s http://ollama.apps.openshift.lab/api/tags | jq .
{
"models": [
{
"name": "falcon:7b-instruct",
"modified_at": "2023-10-26T14:58:39.763638601Z",
"size": 4210994570,
"digest": "4280f7257e73108cddb43de89eb9fa28350a21aaaf997b5935719f9de0281563"
},
{
"name": "llama2:13b",
"modified_at": "2023-10-24T01:12:40.264559502Z",
"size": 7365847104,
"digest": "b3f03629d9a68f2660285cd86d786079c38a56a235032557ab9ba170786f0cf3"
},
{
"name": "llama2:latest",
"modified_at": "2023-10-24T01:11:14.791578247Z",
"size": 3825819519,
"digest": "fe938a131f40e6f6d40083c9f0f430a515233eb2edaa6d72eb85c50d64f2300e"
},
{
"name": "mistral:7b-instruct-q5_K_M",
"modified_at": "2023-10-24T01:10:09.083592659Z",
"size": 5131409362,
"digest": "1041946669e8163f5e8f2a505ea9199f7065eb2c3391e1b08f331005c026fd12"
},
{
"name": "mistral:latest",
"modified_at": "2023-10-26T19:55:56.073227234Z",
"size": 4108916688,
"digest": "8aa307f73b2622af521e8f22d46e4b777123c4df91898dcb2e4079dc8fdf579e"
}
]