/jetson-containers

Machine Learning Containers for NVIDIA Jetson and JetPack-L4T

Primary LanguagePythonMIT LicenseMIT

a header for a software project about building containers for AI and machine learning

Machine Learning Containers for Jetson and JetPack

l4t-pytorch l4t-tensorflow l4t-ml l4t-diffusion l4t-text-generation

Modular container build system that provides various AI/ML packages for NVIDIA Jetson 🚀🤖
ML pytorch tensorflow onnxruntime deepstream jupyterlab stable-diffusion
LLM NanoLLM transformers text-generation-webui ollama llama.cpp exllama llava awq AutoGPTQ MLC optimum nemo
L4T l4t-pytorch l4t-tensorflow l4t-ml l4t-diffusion l4t-text-generation
VIT NanoOWL NanoSAM Segment Anything (SAM) Track Anything (TAM) clip_trt
CUDA cupy cuda-python pycuda numba cudf cuml
Robotics ros ros2 opencv:cuda realsense zed oled
RAG llama-index langchain jetrag NanoDB FAISS RAFT
Audio whisper whisper_trt piper riva audiocraft voicecraft
Smart Home homeassistant-core homeassistant-base wyoming-whisper wyoming-openwakeword wyoming-piper wyoming-assist-microphone

See the packages directory for the full list, including pre-built container images for JetPack/L4T.

Using the included tools, you can easily combine packages together for building your own containers. Want to run ROS2 with PyTorch and Transformers? No problem - just do the system setup, and build it on your Jetson:

$ jetson-containers build --name=my_container pytorch transformers ros:humble-desktop

There are shortcuts for running containers too - this will pull or build a l4t-pytorch image that's compatible:

$ jetson-containers run $(autotag l4t-pytorch)

jetson-containers run launches docker run with some added defaults (like --runtime nvidia, mounted /data cache and devices)
autotag finds a container image that's compatible with your version of JetPack/L4T - either locally, pulled from a registry, or by building it.

If you look at any package's readme (like l4t-pytorch), it will have detailed instructions for running it.

Changing CUDA Versions

You can rebuild the container stack for different versions of CUDA by setting the CUDA_VERSION variable:

CUDA_VERSION=12.4 jetson-containers build transformers

It will then go off and either pull or build all the dependencies needed, including PyTorch and other packages that would be time-consuming to compile. There is a Pip server that caches the wheels to accelerate builds. You can also request specific versions of cuDNN, TensorRT, Python, and PyTorch with similar environment variables like here.

Documentation

Check out the tutorials at the Jetson Generative AI Lab!

Getting Started

Refer to the System Setup page for tips about setting up your Docker daemon and memory/storage tuning.

# install the container tools
git clone https://github.com/dusty-nv/jetson-containers
bash jetson-containers/install.sh

# automatically pull & run any container
jetson-containers run $(autotag l4t-pytorch)

Or you can manually run a container image of your choice without using the helper scripts above:

sudo docker run --runtime nvidia -it --rm --network=host dustynv/l4t-pytorch:r36.2.0

Looking for the old jetson-containers? See the legacy branch.

Gallery

Multimodal Voice Chat with LLaVA-1.5 13B on NVIDIA Jetson AGX Orin (container: NanoLLM)


Interactive Voice Chat with Llama-2-70B on NVIDIA Jetson AGX Orin (container: NanoLLM)


Realtime Multimodal VectorDB on NVIDIA Jetson (container: nanodb)


NanoOWL - Open Vocabulary Object Detection ViT (container: nanoowl)

Live Llava on Jetson AGX Orin (container: NanoLLM)

Live Llava 2.0 - VILA + Multimodal NanoDB on Jetson Orin (container: NanoLLM)

Small Language Models (SLM) on Jetson Orin Nano (container: NanoLLM)

Realtime Video Vision/Language Model with VILA1.5-3b (container: NanoLLM)