ollama-voice-jetson: A Python repository from rhysdg

Ollama Voice: Jetson Xavier Edition

Jetson (Xavier series) compatible LLM based assistant and robot control package
Explore the docs »

Report Bug . Request Feature

About the Project
- Built With
- The Story so Far
Getting Started
Proposed Updates
Contact

About The Project

Built With

Hardware

The Story So Far

This project came into fruition through an excited bit of research into the increasing ability to deploy LLMs on edge devices. It owes its inspiration and initial structure to a number of initiatives - most notably ollama reference above but also:

A number of changes have been made or are in progress that diverge from these original projects:

pyttsx3 has been swapped out for Piper TTS - while nsss and Sapi5 have great voice support for mac and Windows respectively, espeak leaves much to be desired for Linux. Piper tts is not only fast but has outsanding pronounciation and quality in general - you'll find all dependencies and a sample voice is installed by this project
I'm using gemma 2b rather than llama here with great results on the board. Both latency and mobile battery power
Finally you'll notice that support is slowly coming to a halt for Xavier devices with Orin in the picture. This project is both an attenpt to keep the series alive as they're still incredibly powerful - quantised ollama models run with impressive low latency - but also an alternative approach to a dockerised environment for those who are both new to docker and docker as a tool to solve Jetson compatibility issues.

Given the last point the project assumes that you've made it as far as Jetpack 5.1.4 - although Jetpack 6 support would be great once I can get my hands on an Orin device!

Getting Started:

We're assuming here that you've have a board ready to go - check out the hardware link above for the Xavier NX I'm using
If you're not set up yet you'll need to to install jetpack 5.1.4 with an SD card or a NVMe SSD by folling the official guide here
If you're headed down the NVMe road you can also use the the following bug fix branch from the fantastic Jetson Hacks in order to expand the partition size of your root on installation
All installation scripts are self contained and debugged for this specific setup but note that you'll need to ensure you're in a Python 3.9 virtual environment rather than the stock Python 3.8 - this is due to dependency conflicts at Piper-TTS. All dependencies at install.sh include any relevant 3.9 wheels as a result
Download the pre-built torch 2.1.0 wheel for Python 3.9 at the following Google Drive link and add to your project root
Then simply run
```
sh install.sh
```

Customisation:

You'll notice a number of customisable options at the installation script
- The whisper model downloaded to the whisper subfolder is the stock base.pt
- The onnx voice available via the script is en_GB-alba-medium.onnx this is also easily changed by dropping a new voice into voices
assistant.yaml also points to a number of project specific parrameters:
- Notice that my ollama model is mini-b named after a robot I'm bulding - check out ollama's customize prompt section for more on this

Notebooks

Coming soon

Tools and Scripts

coming soon

Testing

Jetpack 5.1.4 - Xavier NX Development Kit - passing
Likely appicable to an AGX Xavier too - I have one available - testing results shortly

Supplementary Data

coming soon

Latency benchmarks