Is it possible to use docker environment variables for substitution in static configuration files?
Closed this issue · 12 comments
My aim is to use AutoKuma to store and manage my Uptime-Kuma configuration, which would be kept in a repository.
Part of this GitOps approach is to avoid storing secrets (passwords, tokens, other sensitive information) in a repo but instead using environment files, secrets files, key vaults, or other systems to reference the sensitive information. A good example of a simple implementation of this is the expand-env config option for loki. It is also available in frigate for certain values.
The directory containing my autokuma compose, environment, and configuration files would look something like this:
/srv/autokuma/compose.yaml
services:
autokuma:
image: ghcr.io/bigboot/autokuma:master
container_name: ${CONTAINER_NAME1}
hostname: ${HOSTNAME1}
domainname: ${DOMAINNAME}
environment:
- AUTOKUMA__DOCKER__ENABLED=${AUTOKUMA__DOCKER__ENABLED}
- AUTOKUMA__STATIC_MONITORS=${AUTOKUMA__STATIC_MONITORS}
- AUTOKUMA__LOG_DIR=${AUTOKUMA__LOG_DIR}
- AUTOKUMA__ON_DELETE=${AUTOKUMA__ON_DELETE}
- AUTOKUMA__KUMA__URL=${AUTOKUMA__KUMA__URL}
- AUTOKUMA__KUMA__USERNAME=${AUTOKUMA__KUMA__USERNAME}
- AUTOKUMA__KUMA__PASSWORD=${AUTOKUMA__KUMA__PASSWORD}
- AUTOKUMA__KUMA__CALL_TIMEOUT=${AUTOKUMA__KUMA__CALL_TIMEOUT}
- AUTOKUMA__KUMA__CONNECT_TIMEOUT=${AUTOKUMA__KUMA__CONNECT_TIMEOUT}
volumes:
- ${DIRECTORY_DATA}:/data
- ${DIRECTORY_STATIC}:/autokuma/static
- ${DIRECTORY_LOGS}:/autokuma/logs
- /etc/timezone:/etc/timezone:ro
restart: unless-stopped
/srv/autokuma/.env.example
# Host specifics
CONTAINER_NAME1=autokuma
CONTAINER_PORT1=
HOSTNAME1=autokuma
DOMAINNAME=example.com
# Directory locations
DIRECTORY_DATA=/srv/autokuma/data
DIRECTORY_STATIC=/srv/autokuma/static
DIRECTORY_LOGS=/srv/autokuma/logs
# Container specifics
AUTOKUMA__DOCKER__ENABLED=false
AUTOKUMA__STATIC_MONITORS=/autokuma/static
AUTOKUMA__LOG_DIR=/autokuma/logs
AUTOKUMA__ON_DELETE=keep
AUTOKUMA__KUMA__URL=http://192.168.1.93:3002
AUTOKUMA__KUMA__USERNAME=[username]
AUTOKUMA__KUMA__PASSWORD=[password]
AUTOKUMA__KUMA__CALL_TIMEOUT=15
AUTOKUMA__KUMA__CONNECT_TIMEOUT=30
# Container secrets
AUTOKUMA__MYSQL_USERNAME=[mysql-username]
AUTOKUMA__MYSQL_PASSWORD=[mysql-password]
AUTOKUMA__POSTGRES_USERNAME=[postgres-username]
AUTOKUMA__POSTGRES_PASSWORD=[postgres-password]
AUTOKUMA__TELEGRAM_BOT_TOKEN=[telegram-bot-token]
AUTOKUMA__TELEGRAM_CHAT_ID=[telegram-chat-id]
/srv/autokuma/static/database.json
[
{
"type": "group",
"name": "databases",
"interval": 120,
"max_retries": 2,
"retry_interval": 120,
"description": "All monitors for databases"
},
{
"type": "postgres",
"name": "teslamate-db.postgres",
"parent_name": "static/database[0]",
"database_connection_string": "postgres://${AUTOKUMA__POSTGRES_USERNAME}:${AUTOKUMA__POSTGRES_PASSWORD}@192.168.1.93:5432/teslamate",
"database_query": "SELECT datname FROM pg_database WHERE datistemplate = false;"
},
{
"type": "mysql",
"name": "photoprism-db.mariadb",
"parent_name": "static/database[0]",
"database_connection_string": "mysql://${AUTOKUMA__MYSQL_USERNAME}@192.168.1.93:3310/photoprism",
"radius_password": "${AUTOKUMA__MYSQL_PASSWORD}",
"database_query": "SHOW DATABASES;"
}
]
/srv/autokuma/static/notification.json
{
"type": "notification",
"active": true,
"isDefault": true,
"config":
{
"type": "telegram",
"active": true,
"isDefault": true,
"name": "telegram",
"telegramBotToken": "${AUTOKUMA__TELEGRAM_BOT_TOKEN}",
"telegramChatID": ${AUTOKUMA__TELEGRAM_CHAT_ID},
"telegramMessageThreadID": "6",
"userId": 1
}
}
The steps to stand up the AutoKuma instance would be:
- cd /srv/autokuma
- cp .env.example .env
- nano .env
- Replace all [substitution-values] with sensitive data
- docker compose up --detach
Then the variable substitution would do the work.
Is this something that AutoKuma can already do? If not, could it be added to the roadmap?
Hi, this should be possible, static monitors allow using tera templates, which has a built in get_env function. So something like this should work.
{
"type": "postgres",
"name": "teslamate-db.postgres",
"parent_name": "static/database[0]",
"database_connection_string": "postgres://{{ get_env(name=\"AUTOKUMA__POSTGRES_USERNAME\" }}:{{ get_env(name=\"AUTOKUMA__POSTGRES_PASSWORD\" }}@192.168.1.93:5432/teslamate",
"database_query": "SELECT datname FROM pg_database WHERE datistemplate = false;"
}Now that I think about this, I'll probably restrict access to variables starting with AUTOKUMA__ENV__ or something in the future for security reasons.
This is what I've got:
{
"type": "postgres",
"name": "teslamate-db.postgres",
"parent_name": "static/database[0]",
"database_connection_string": "postgres://{{ get_env(name=\"AUTOKUMA__POSTGRES_USERNAME\" }}:{{ get_env(name=\"AUTOKUMA__POSTGRES_PASSWORD\" }}@192.168.1.93:5432/teslamate",
"database_query": "SELECT datname FROM pg_database WHERE datistemplate = false;"
}
This is what I used:
# Container secrets
AUTOKUMA__MYSQL_USERNAME=[mysql-username]
AUTOKUMA__MYSQL_PASSWORD=[mysql-password]
AUTOKUMA__POSTGRES_USERNAME=[postgres-username]
AUTOKUMA__POSTGRES_PASSWORD=[postgres-password]
This is what I get:
autokuma | _ _ __
autokuma | /\ | | | |/ /
autokuma | / \ _ _ | |_ ___ | ' / _ _ _ __ ___ __ _
autokuma | / /\ \ | | | || __| / _ \ | < | | | || '_ ` _ \ / _` |
autokuma | / ____ \ | |_| || |_ | (_) || . \ | |_| || | | | | || (_| |
autokuma | /_/ \_\ \__,_| \__| \___/ |_|\_\ \__,_||_| |_| |_| \__,_|
autokuma | v0.8.0-bc06decc
autokuma | WARN [kuma_client::util] [id-database.json] Error while trying to parse labels: --> 1:58
autokuma | |
autokuma | 1 | database_connection_string = "postgres://{{ get_env(name=\"AUTOKUMA__POSTGRES_USERNAME\" }}:{{ get_env(name=\"AUTOKUMA__POSTGRES_PASSWORD\" }}@192.168.1.93:5432/teslamate"
autokuma | | ^---
autokuma | |
autokuma | = expected a value that can be negated or an array of values
autokuma | Context: None
autokuma | WARN [kuma_client::util] [id-database.json] Error while trying to parse labels: --> 1:55
autokuma | |
autokuma | 1 | database_connection_string = "mysql://{{ get_env(name=\"AUTOKUMA__POSTGRES_USERNAME\" }}@192.168.1.93:3310/photoprism"
autokuma | | ^---
autokuma | |
autokuma | = expected a value that can be negated or an array of values
autokuma | Context: None
Removed the escape slashes and got the following:
autokuma | v0.8.0-bc06decc
autokuma | WARN [kuma_client::util] [/autokuma/static/id-database.json] Unable to deserialize: expected `,` or `}` at line 14 column 68
Looks like the escape slashes are important, but I can't work out how to get the templated get_env pull to work.
I realised I didn't add the environment values in .env into my compose.yaml file, so I did that and reupped the container.
Running docker compose config now shows the environment variables are available to the running container, including:
AUTOKUMA__POSTGRES_PASSWORD: [postgres-password]
AUTOKUMA__POSTGRES_USERNAME: [postgres-username]
But the same error is occurring:
autokuma | v0.8.0-bc06decc
autokuma | WARN [kuma_client::util] [id-database.json] Error while trying to parse labels: --> 1:58
autokuma | |
autokuma | 1 | database_connection_string = "postgres://{{ get_env(name=\"AUTOKUMA__POSTGRES_USERNAME\" | string }}:{{ get_env(name=\"AUTOKUMA__POSTGRES_PASSWORD\" | string }}@192.168.1.93:5432/teslamate"
autokuma | | ^---
autokuma | |
autokuma | = expected a value that can be negated or an array of values
autokuma | Context: None
autokuma | WARN [kuma_client::util] [id-database.json] Error while trying to parse labels: --> 1:55
autokuma | |
autokuma | 1 | database_connection_string = "mysql://{{ get_env(name=\"AUTOKUMA__POSTGRES_USERNAME\" | string }}@192.168.1.93:3310/photoprism"
autokuma | | ^---
autokuma | |
autokuma | = expected a value that can be negated or an array of values
autokuma | Context: None
I'm sorry yes the " seems to be problematic going through the whole templating machinery, you can use ' or ` instead:
{
"type": "postgres",
"name": "teslamate-db.postgres",
"parent_name": "static/database[0]",
"database_connection_string": "postgres://{{ get_env(name='AUTOKUMA__POSTGRES_USERNAME' }}:{{ get_env(name='AUTOKUMA__POSTGRES_PASSWORD' }}@192.168.1.93:5432/teslamate",
"database_query": "SELECT datname FROM pg_database WHERE datistemplate = false;"
}Makes sense that the double quote would mess things up! Got this now:
autokuma | v0.8.0-bc06decc
autokuma | WARN [kuma_client::util] [id-database.json] Error while trying to parse labels: --> 1:88
autokuma | |
autokuma | 1 | database_connection_string = "postgres://{{ get_env(name='AUTOKUMA__POSTGRES_USERNAME' }}:{{ get_env(name='AUTOKUMA__POSTGRES_PASSWORD' }}@192.168.1.93:5432/teslamate"
autokuma | | ^---
autokuma | |
autokuma | = expected `or`, `and`, `not`, `<=`, `>=`, `<`, `>`, `==`, `!=`, or a filter
autokuma | Context: None
autokuma | WARN [kuma_client::util] [id-database.json] Error while trying to parse labels: --> 1:82
autokuma | |
autokuma | 1 | database_connection_string = "mysql://{{ get_env(name='AUTOKUMA__MYSQL_USERNAME' }}@192.168.1.93:3310/photoprism"
autokuma | | ^---
autokuma | |
autokuma | = expected `or`, `and`, `not`, `<=`, `>=`, `<`, `>`, `==`, `!=`, or a filter
autokuma | Context: None
I added back the cast to string:
autokuma | v0.8.0-bc06decc
autokuma | WARN [kuma_client::util] [id-database.json] Error while trying to parse labels: --> 1:97
autokuma | |
autokuma | 1 | database_connection_string = "postgres://{{ get_env(name='AUTOKUMA__POSTGRES_USERNAME' | string }}:{{ get_env(name='AUTOKUMA__POSTGRES_PASSWORD' }}@192.168.1.93:5432/teslamate"
autokuma | | ^---
autokuma | |
autokuma | = expected `or`, `and`, `not`, `<=`, `>=`, `<`, `>`, `==`, `!=`, or a filter
autokuma | Context: None
Still nothing.
Sorry I just realized the closing ) is missing, the string filter is also not needed as get_env returns a string anyway:
{
"type": "postgres",
"name": "teslamate-db.postgres",
"parent_name": "static/database[0]",
"database_connection_string": "postgres://{{ get_env(name='AUTOKUMA__POSTGRES_USERNAME') }}:{{ get_env(name='AUTOKUMA__POSTGRES_PASSWORD') }}@192.168.1.93:5432/teslamate",
"database_query": "SELECT datname FROM pg_database WHERE datistemplate = false;"
}In hindsight it's obvious about the closing bracket!
Progress!
autokuma | v0.8.0-14d7608e
autokuma | WARN [kuma_client::util] [id-database.json] Error while trying to parse labels:
autokuma | Caused by: Failed to render 'database_connection_string = "postgres://{{ get_env(name='AUTOKUMA__POSTGRES_USERNAME') }}:{{ get_env(name='AUTOKUMA__POSTGRES_PASSWORD') }}@192.168.1.93:5432/teslamate"
autokuma | database_query = "SELECT datname FROM pg_database WHERE datistemplate = false;"
autokuma | name = "teslamate-db.postgres"
autokuma | parent_name = "database[0]"
autokuma | type = "postgres"'
autokuma | Caused by: Function call 'get_env' failed
autokuma | Caused by: Access to environment variable `AUTOKUMA__POSTGRES_USERNAME` is not allowed
We have this for:
- AUTOKUMA__POSTGRES_USERNAME
- AUTOKUMA__MYSQL_PASSWORD
- AUTOKUMA__TELEGRAM_BOT_TOKEN
It may not appear for the others as they're part of the set being parsed.
I added an extra underscore to the variables names just in case this was required:
# Container secrets
AUTOKUMA__MYSQL__USERNAME=[mysql-username]
AUTOKUMA__MYSQL__PASSWORD=[mysql-password]
AUTOKUMA__POSTGRES__USERNAME=[postgres-username]
AUTOKUMA__POSTGRES__PASSWORD=[postgres-password]
AUTOKUMA__TELEGRAM__BOT__TOKEN=[telegram-bot-token]
AUTOKUMA__TELEGRAM__CHAT__ID=[telegram-chat-id]
Previously there was only __ after AUTOKUMA. This didn't make a difference:
autokuma | Caused by: Function call 'get_env' failed
autokuma | Caused by: Access to environment variable `AUTOKUMA__POSTGRES__USERNAME` is not allowed
autokuma | Caused by: Function call 'get_env' failed
autokuma | Caused by: Access to environment variable `AUTOKUMA__MYSQL__USERNAME` is not allowed
Yes I've restricted access like I said above, you can either rename your variables or allow access to all environment variables:
Understood. Renamed the variables:
- AUTOKUMA__ENV__MYSQL__USERNAME=${AUTOKUMA__MYSQL__USERNAME}
- AUTOKUMA__ENV__MYSQL__PASSWORD=${AUTOKUMA__MYSQL__PASSWORD}
- AUTOKUMA__ENV__POSTGRES__USERNAME=${AUTOKUMA__POSTGRES__USERNAME}
- AUTOKUMA__ENV__POSTGRES__PASSWORD=${AUTOKUMA__POSTGRES__PASSWORD}
- AUTOKUMA__ENV__TELEGRAM__BOT__TOKEN=${AUTOKUMA__TELEGRAM__BOT__TOKEN}
- AUTOKUMA__ENV__TELEGRAM__CHAT__ID=${AUTOKUMA__TELEGRAM__CHAT__ID}
Made sure the .env file matched:
# Container secrets
AUTOKUMA__ENV__MYSQL__USERNAME=[mysql-username]
AUTOKUMA__ENV__MYSQL__PASSWORD=[mysql-password]
AUTOKUMA__ENV__POSTGRES__USERNAME=[postgres-username]
AUTOKUMA__ENV__POSTGRES__PASSWORD=[postgres-password]
AUTOKUMA__ENV__TELEGRAM__BOT__TOKEN=[telegram-bot-token]
AUTOKUMA__ENV__TELEGRAM__CHAT__ID=[telegram-chat-id]
Used the correct variable names on the static files:
{
"type": "postgres",
"name": "teslamate-db.postgres",
"parent_name": "id-database[0]",
"database_connection_string": "postgres://{{ get_env(name='AUTOKUMA__ENV__POSTGRES__USERNAME') }}:{{ get_env(name='AUTOKUMA__ENV__POSTGRES__PASSWORD') }}@192.168.1.93:5432/teslamate",
"database_query": "SELECT datname FROM pg_database WHERE datistemplate = false;"
},
Everything was configured, but the variables did not get imported.
teslamate:
photoprism:
telegram:
Just to make sure:
- AUTOKUMA__ENV__MYSQL__USERNAME=${AUTOKUMA__MYSQL__USERNAME} - AUTOKUMA__ENV__MYSQL__PASSWORD=${AUTOKUMA__MYSQL__PASSWORD} - AUTOKUMA__ENV__POSTGRES__USERNAME=${AUTOKUMA__POSTGRES__USERNAME} - AUTOKUMA__ENV__POSTGRES__PASSWORD=${AUTOKUMA__POSTGRES__PASSWORD} - AUTOKUMA__ENV__TELEGRAM__BOT__TOKEN=${AUTOKUMA__TELEGRAM__BOT__TOKEN} - AUTOKUMA__ENV__TELEGRAM__CHAT__ID=${AUTOKUMA__TELEGRAM__CHAT__ID}AUTOKUMA__ENV__MYSQL__USERNAME=[mysql-username]
AUTOKUMA__ENV__MYSQL__PASSWORD=[mysql-password]
AUTOKUMA__ENV__POSTGRES__USERNAME=[postgres-username]
AUTOKUMA__ENV__POSTGRES__PASSWORD=[postgres-password]
AUTOKUMA__ENV__TELEGRAM__BOT__TOKEN=[telegram-bot-token]
AUTOKUMA__ENV__TELEGRAM__CHAT__ID=[telegram-chat-id]
The names do not match, is this just a copy-paste error?
Well that's the issue with trusting global search and replace in Visual Studio Code when I was rushing to get out the door this morning! Fixed, and now the environment variables are coming through.
Just need to work out what database_query should actually be to pass the query, I'll try databaseQuery... Which worked.