/run-llm-on-raspberry-pi

Instructions on how to run LLMs on Raspberry PI

How to run LLM (mistral 7b) on Raspberry Pi 5

How to run LLM (mistral 7b) on Raspberry Pi 5

Step-by-Step Guide on how to run Large Language Model on a Raspberry Pi 5 (might work on 4 too, haven't tested it yet)

Table of Content:

  1. Prerequisite
  2. Setup Raspberry Pi
  3. Option 1: Run LLMs using Ollama
  4. Option 2: Run LLMs using Llama.cpp
  5. Extra Resoucres

Prerequisite

Setup Raspberry Pi (Headless-setup)

You can also follow along this YouTube video instead.

  1. Connect the SD card to your laptop
  2. Download Raspberry Pi OS (bootloader): https://www.raspberrypi.com/software/
  3. Run it, and you should see: screenshot1.png
    • "Choose Device" - choose Raspberry Pi 5
    • OS, choose the latest (64bit is the recommended)
    • "Choose Storage" - choose the inserted SD card
  4. Now click next, and it will ask you if you want to edit the settings, click "Edit settings" screenshot2.png
  5. Configure screenshot3.png
    • enable hostname and set it to raspberrypi.local
    • Set username and password you will remember, we will use them shortly
    • Enable "Configure Wireless LAN" and add your wifi name and password
    • Click save, and contiue. it will take a few minutes to write everything to the SD
  6. Insert the SD card to your raspberry pi, and connect it to the electricity
  7. SSH into the Raspberry PI:
ssh ssh <YOUR_USERNAME>@raspberrypi.local

Option 1: Run LLMs using Ollama

  1. Install Ollama:
curl -fsSL https://ollama.com/install.sh | sh
  1. Download & Run Mistral model:
ollama run mistral

That is it!

Option 2: Run LLMs using llama.cpp:

  1. Install:
sudo apt update && sudo apt install git g++ wget build-essential
  1. Download llama.cpp repo:
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
  1. Compile:
make -j
  1. Download Mistral model:
cd models
wget https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/resolve/main/mistral-7b-v0.1.Q4_K_S.gguf)
  1. Go back to repo root folder, and run:
cd ..
./main -m models/mistral-7b-v0.1.Q4_K_S.gguf -p "Whatsup?" -n 400 -e

That is it!

Extra Resoucres: