Improve docker containers and build/setup process.
Opened this issue · 3 comments
The current docker container setup was thrown together to get something up and running. When encountering things that wouldn't work properly (especially for the single-container DockerFile, where I did some head-scratching environment stuff, as a sort of beating the container into submission to do my bidding). Docker is not my area of expertise, and due to this, the current container set up leaves a lot to be desired. I am opening this issue, to clarify my goals for the project, and how the envision the containers working.
Goals
- Simple initial setup for users. Above all, this is the most important goal I have for the project, and is probably the biggest issue at the moment. The Twitch APIs are a huge hassle to deal with- I want TAU to be a simple to use solution to that problem. And while many developers have some experience with using docker, there are many, like myself, who have only used it for very specific cases. This will require 2 components- making the docker setup as simple as possible, and providing clear setup documentation (see issue #8).
- Running locally using properly siloed services (likely using docker-compose and multiple containers). See the list of services that need to be run below. Having processes running in separate containers definitely helps with troubleshooting and development, and is definitely best practice. However...
- A single container build for use as a cloud service. There are several TAU users who are running it as a cloud service, so that their TAU instances can be capturing events such as follows and subscriptions, even when they're offline. Additionally, with the new streamer panel for discord bots (fires off online/offline events for a list of streamers- useful for things like discord bots), the use of a remote/constantly running TAU instance is important. The services I know of that currently have TAU instances running are Render.com, Azure containers, and Vultr. In all three cases, pricing is based on a per-container basis, and since the resources needed to run TAU are quite low, a single container build is much more cost effective.
- VSCode .devcontainers for developing/debugging TAU in VSCode. This one should be easy- with a proper docker-compose file, we can just point the devcontainer setup properly, and it should just work.
Required Services
In order for TAU to run, we need the following services:
- daphane/django webserver- serves up the client-facing websocket. Requires Python 3.8 image.
- django Twitch websocket/webhook connection management client. Requires Python 3.8 image.
- django Twitch IRC connection management client. Requires Python 3.8 image.
- database- either Postgres (requires Postgres image) or sqlite3 (no additional image/service, lives in container(s) for 1,2,3)
- ngrok (optional)- Uses python ngrok module, requires no additional service/image. (lives in container(s) for 1,2,3)
- Redis server for django channels. Requires Redis image.
Currently 1, 2, and 3 share a single container with the three processes being managed with Supervisord.
For the single-DockerFile build, a Redis image is added to the build and the Redis process is managed with supervisord. Additionally, sqlite3 is used in favor of postgres to eliminate the database container.
Related Issues and PRs
Issues
- #8 Improve documentation for setup process, specifically for single-container setup.
- #30 Thoughts on additional documentation needs
- #41 Lock image versions for postgres and Redis
PRs
- #36 Add support for external Redis provider such as the free tier at RedisLabs. This would primarily be used for the single-container (cloud) deployment, and let us remove Redis from that container, while not requiring a second container to be run.
- #39 Refactoring docker-compose file to follow docker best practices more closely. Uses defaults for some defaultable environment variables (making setting up the .env file easier). Also changes how supervisord is installed and run, outputting logs to stdout rather than a file, so that they can be read through
docker logs
Thinking about some smart defaults to simplify the .env file, I think we can simplify the most basic to this:
TWITCH_APP_ID=<MUST BE PROVIDED>
TWITCH_CLIENT_SECRET=<MUST BE PROVIDED>
TWITCH_WEBHOOK_SECRET=<PRE POPULATED RANDOM STRING>
PORT=8000
POSTGRES_PW=<PRE POPULATED RANDOM STRING>
DJANGO_DB_PW=<PRE POPULATED RANDOM STRING>
DJANGO_SECRET_KEY=<PRE POPULATED RANDOM STRING>
The sample .env would provide pre-populated random strings for POSTGRES_PW
, DJANGO_DB_PW
, DJANGO_SECRET_KEY
, and TWITCH_WEBHOOK_SECRET
however the TWITCH_APP_ID
, and TWITCH_CLIENT_SECRET
need to be provided from the twitch Dev portal. The following defaults would be set up in the docker-compose file, and could be overridden by providing them in the .env file:
# Django Defaults
DJANGO_DB_TYPE=postgres
DJANGO_DB=tau_db
DJANGO_DB_USER=tau_db
DJANGO_DB_HOST=db
DJANGO_CONFIGURATION=Local
# Network Defaults
PROTOCOL=http:
PUBLIC_URL=localhost
#Ngrok Defaults
USE_NGROK=True
And there would be the following addtional parameters that would typically be used only in cloud deployments (and by default are not set at all)
# Additional Redis settings
REDIS_ENDPOINT=some.remote.redis.server.com:port
REDIS_PW=<password for said redis server>
# Addtional Ngrok settings
NGROK_TOKEN=<ngrok token for non-anonymous account>
By using smart defaults, and not requiring enviornment variables for unused features, the complexity will be significantly reduced for the basic .env file which should work for most local instances.
@FiniteSingularity see PR #45 . I'm sure this would need to get updated for the smart defaults you're thinking of.