Docker image for the Text Generation Web UI: A Gradio web UI for Large Language Models. Supports Transformers, AWQ, GPTQ, llama.cpp (GGUF), Llama models

Note

The legacy APIs no longer work with the latest version of the Text Generation Web UI. They were deprecated in November 2023 and have now been completely removed. If you want to use the LEGACY APIs, please set the image tag to 1.9.5. You will also have to add port 6000 for the legacy REST API and/or port 6005 for the legacy Websockets API.

Installs

Ubuntu 22.04 LTS
CUDA 12.1.1
Python 3.11.9
Text Generation Web UI v1.10.1
Torch 2.2.2
Jupyter Lab
code-server
runpodctl
OhMyRunPod
RunPod File Uploader
croc
rclone
speedtest-cli
screen
tmux

Available on RunPod

This image is designed to work on RunPod. You can use my custom RunPod template to launch it on RunPod.

Building the Docker image

Note

You will need to edit the docker-bake.hcl file and update REGISTRY_USER, and RELEASE. You can obviously edit the other values too, but these are the most important ones.

# Clone the repo
git clone https://github.com/ashleykleynhans/text-generation-docker.git

# Log in to Docker Hub
docker login

# Build the image, tag the image, and push the image to Docker Hub
cd text-generation-docker
docker buildx bake -f docker-bake.hcl --push

# Same as above but customize registry/user/release:
REGISTRY=ghcr.io REGISTRY_USER=myuser RELEASE=my-release docker buildx \
    bake -f docker-bake.hcl --push

Running Locally

Install Nvidia CUDA Driver

Start the Docker container

docker run -d \
  --gpus all \
  -v /workspace \
  -p 3000:3001 \
  -p 5000:5001 \
  -p 7777:7777 \
  -p 8888:8888 \
  -p 2999:2999 \
  -e VENV_PATH="/workspace/venvs/text-generation-webui" \
  ashleykza/oobabooga:latest

You can obviously substitute the image name and tag with your own.

Ports

Connect Port	Internal Port	Description
3000	3001	Text Generation Web UI
5000	5001	Open AI Compatible API
7777	7777	Code Server
8888	8888	Jupyter Lab
2999	2999	RunPod File Uploader

Environment Variables

Variable	Description	Default
VENV_PATH	Set the path for the Python venv for the app	/workspace/venvs/text-generation-webui
JUPYTER_LAB_PASSWORD	Set a password for Jupyter lab	not set - no password
DISABLE_AUTOLAUNCH	Disable Web UI from launching automatically	(not set)
DISABLE_SYNC	Disable syncing if using a RunPod network volume	(not set)

Logs

Text Generation Web UI creates a log file, and you can tail the log instead of killing the service to view the logs.

Application	Log file
Text Generation Web UI	/workspace/logs/textgen.log

For example:

tail -f /workspace/logs/textgen.log

Community and Contributing

Pull requests and issues on GitHub are welcome. Bug fixes and new features are encouraged.

ashleykleynhans/text-generation-docker