Jetson Copilot

Jetson Copilot is a reference application for a local AI assistant.

It demonstrates two things;

Running open-source LLMs (large language model) on device
Augmenting the LLM to have access to your locally indexed knowledge (RAG, retrieval-augmented generation)

Important

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

OSS	Repo URL	Usage
Ollama	GitHub	To host and run LLMs locally, including embedding models for building index from documents
LlamaIndex	GitHub	Data framework for LLM, used mainly to realize RAG pipeline.
Streamlit	GitHub	Python library to create an interactive web app

🏃 Getting started

First time setup

If this is your first time to run Jetson Copilot on your Jetson, first run setup_environment.sh to ensure you have all the necessary software installed and the environment set up.

git clone https://github.com/NVIDIA-AI-IOT/jetson-copilot/
cd jetson-copilot
./setup_environment.sh

It will install the following, if not yet.

Chromium web browser
Docker

Important

It also adds the current user into docker group using newgrp command.

Check to see if you can just issue the following command (without using sudo).

docker ps

If you get permission denied error, exit from the session (by executing exit command), reboot if possible (so that it will take effect in any sessions), and log/ssh back in.
You should now be able to issue docker ps.

How to start Jetson Copilot

cd jetson-copilot
./launch_jetson_copilot.sh

launch_cui.mp4

This will start a Docker container and start a Ollama server and Streamlit app inside the container. It will shows the URL on the console in order to access the web app hosted on your Jetson.

With your web browser on Jetson, open the Local URL (localhost). Or on a PC connected on the same network as on your Jetson, access the Network URL.

Local URL: http://localhost:8501
Network URL: http://10.110.50.252:8501

Note

You will need the Internet connection on Jetson when it launches for the first time, as it will pull the container image (and download the default LLM and embedding model when web UI starts for the first time).

When you access the web UI for the first time, it will download the default LLM (llama3) and the embedding model (mxbai-embed-large).

Tip

If you are on Ubuntu Desktop, a frameless Chromium window will pop up to access the web app, to make it look like an independent application. You need to close the window as stopping the container on the console won't shutdown Chromium.

desktop_launch.mp4

📖 How to use Jetson Copilot

0. Interact with the plain Llama3 (8b)

llm_question_OS-and-version.mp4

You can use Jetson Copilot just to interact with a LLM without enabling RAG feature.

By default, Llama3 (8b) model is downloaded when running for the first time and use as the default LLM.

You will be surprized how much a model like Llama3 is capable, but may soon find limitations as it does not have information prior to its cutoff date nor know anything about your specific subject matter.

1. Ask Jetson related question using pre-built index

rag_usb-device-mode.mp4

On the side panel, you can toggle "Use RAG" on to enable RAG pipeline.
The LLM will have an access to a custom knowledge/index that is selected under "Index".

As a sample, a pre-build index "_L4T_README" is provided.
This is built on all the README text files that supplied in the "L4T-README" folder on the Jetson desktop.

It is mounted as /media/<USER_NAME>/L4T-README/ once you execute udisksctl mount -b /dev/disk/by-label/L4T-README.

You can ask questions like:

What IP address does Jetson gets assigned when connected to a PC via a USB cable in USB Device Mode?

2. Build your own index based on your documents

rag_build_index.mp4

You can build your own index based on your local and/or online documents.

First, on the console (or on the desktop) create a directory under Documents directory to store your documents.

cd jetson-copilot
mkdir Documents/Jetson-Orin-Nano
cd Documents/Jetson-Orin-Nano
wget https://developer.nvidia.com/downloads/assets/embedded/secure/jetson/orin_nano/docs/jetson_orin_nano_devkit_carrier_board_specification_sp.pdf

Now back on the web UI, open the side bar, toggle on "Use RAG", then click on "➕Build a new index" to jump to a "Build Index" page.

Give a name for the Index you are to build. (e.g. "JON Carrier Board")
Type in the field and hit Enter key, then it will check and show what path will be created for your index.

And then from the drop select box under "Local documents", select the directory you created and saved your documents in. (e.g. /opt/jetson_copilot/Documents/Jetson-Orin-Nano).

It will show the summary of files found in the selected directory.

If you want to rather only or additionally supply URLs for the online docuemnts to be ingested, fill the text area with one URL per a line.
You can skip this if you are building your index only based on your local documents.

Note

On the sidebar, make sure mxbai-embed-large is selected for the embedding model.

Use of OpenAI embedding models is not well supported and needs more testing.

Finally, hit "Build Index" button.
It will show the progress in the drop-down "status container", so you can check the status by clicking on it.
Once done, it will show the summary of your index and time it took.

You can go back to the home screen to now select the index you just built.

3. Test different LLM or Embedding model

TODO

🏗️ Development

Streamlit based web app is very easy to develop.

On web UI, at the top-right of the screen, choose "Always rerun" to automatically update your app every time you change the source codes.

See Streamlit Documentation for the detail.

Manually run streamlit app inside the container

In case you make more fundamental changes, you can also manually run streamlit app.

cd jetson-copilot
./launch_dev.sh

Once in container;

streamlit run app.py

launch_dev.mp4

🧱 Directory structure

└── jetson-copilot
    ├── launch_jetson_copilot.sh
    ├── setup_environment.sh
    ├── Documents 
    │   └── your_abc_docs
    ├── Indexes
    │   ├── _L4T_README
    │   └── your_abc_index
    ├── logs
    │   ├── container.log
    │   └── ollama.log
    ├── ollama_models
    └── Streamlit_app
        ├── app.py
        ├── build_index.py
        └── download_model.py

Following directories inside the jetson-copilot directory are mounted in the Docker container.

Directory Name	Description
`Documents`	Directory to store your documents to be indexed
`Indexes`	Directory to store pre-built (or built-by-you) indexes for LLM to perform RAG on
`logs`	Directory for the app to store log files
`ollama_models`	Directory for the ollama server to store download models
`stremlit_app`	Directory for Python scripts to make up the web app

💫 Troubleshooting

If you find any issue, please check GitHub Issues of the Jetson Copilot repo and file an issue there.

`permission denied`

If you get "permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock" issue, your current user account in Linux may not be added to the group docker and/or the change is not taking effect yet.

So issue the following commands to ensure you add your user account in the group, and exit from the current session, reboot if possible (so that it will take effect in any sessions), then log/ssh back in.

sudo systemctl restart docker
sudo usermod -aG docker $USER
newgrp docker
exit

You can check if you succeed by issuing docker ps.

`unable to setup input stream`

If you run launch_jetson_copilot.sh and get "unable to setup input stream: unable to set IO streams as raw terminal: input/output error", then you can just use the launch_dev.sh and manually start the streamlit app.

./launch_dev.sh

And once in the container;

streamlit run app.py

Proposed usage

You can use Jetson Copilot in multiple ways.

Useful and tool

Users can easily run an LLM on Jetson without relying on any cloud services.

They may find the AI assistance on some tasks useful, like to find out the right command to use on Linux system. They can even expand the LLM knowledge by building the local index based on their own documents that LLM can access.

Reference for building a custom AI assistant

Developers can use Jetson Copilot as a reference for building their own AI assistant that possesses some specific domain area's or product's knowledge.

⚖️ License

Please see LICENSE file.

📜 Project status

Pushed to public, still partially in development.

TODO

LLM download UI (download_model.py)
Use faiss for Vector DB
Support OpenAI embedding models