When using {{ key }} with missing value and multiple templates consul-template --wait : global quiescence is never reached

[markotitel]

Consul Template version

hashicorp/consul-template:0.33

Configuration

There is no configuration other than parameters provided in the Steps to reproduce section below.

Command

Command is provided in Steps to reproduce section.

Debug output

It is inconvenient to provide output since there is no specific command neither specific template. In Steps to reproduce it is demonstrated what happens with minimal viable config.

Expected behavior

consul-template should reach enabling global quiescence state for rendered templates and --wait parameter should work as expected with Consul KV changes.

When updating Consul KV multiple values for the template command should execute only once.

We should see similar to:

2023-09-28T05:30:26.016Z [DEBUG] (runner) enabling global quiescence for "1d56665649284454fd46ad5c8899f3b8"
2023-09-28T05:30:26.016Z [DEBUG] (runner) enabling global quiescence for "28bf72edc653729a4304301c82492b2d"

Actual behavior

consul-template renders valid templates, skips invalid templates but never gets to global quiescence state therefore --wait param is not usable.

When updating Consul KV multiple values for the template command is executed multiple times.

Steps to reproduce

First example is working example consisting couple of bash commands to create minimal working example.

mkdir /tmp/consul-template-double-deploy/ && cd /tmp/consul-template-double-deploy/

export DOCKER_NETWORK_NAME=consul-template-quiescence-issue

docker network create $DOCKER_NETWORK_NAME

docker run -d --rm --network-alias consul-kv --network $DOCKER_NETWORK_NAME --name consul-kv hashicorp/consul:1.15.4 && sleep 2

docker exec consul-kv consul kv put first_value 100 \
&& docker exec consul-kv consul kv put second_value 200 \
&& docker exec consul-kv consul kv put third_value 300 \
&& docker exec consul-kv consul kv put fourth_value 400

cat <<EOF > tpl_1.ctmpl 
{{ key "first_value" }}
{{ key "second_value" }}
EOF

cat <<EOF > tpl_2.ctmpl 
{{ key "third_value" }}
{{ key "fourth_value" }}
EOF

docker run --rm \
 -e "CONSUL_HTTP_ADDR=consul-kv:8500" \
 -v $(pwd):/tpl \
 --network $DOCKER_NETWORK_NAME \
 --name consul-tpl \
 hashicorp/consul-template:0.33 \
 -log-level=debug \
 --wait=3s \
 -template '/tpl/tpl_1.ctmpl:/tpl/tpl_1.rendered:echo "$(date)" >> /tpl/tpl_1.deployed' \
 -template '/tpl/tpl_2.ctmpl:/tpl/tpl_2.rendered:echo "$(date)" >> /tpl/tpl_2.deployed'

This example reproduces the issue. Only difference is that second template cannot be rendered because of missing KV

Cleanup after the first example

docker kill consul-kv
docker kill consul-tpl
docker network rm consul-template-quiescence-issue
rm -f tpl*

Notice below that fourth_value is missing in the KV.

mkdir /tmp/consul-template-double-deploy/ && cd /tmp/consul-template-double-deploy/

export DOCKER_NETWORK_NAME=consul-template-quiescence-issue

docker network create $DOCKER_NETWORK_NAME

docker run -d --rm --network-alias consul-kv --network $DOCKER_NETWORK_NAME --name consul-kv hashicorp/consul:1.15.4 && sleep 2

docker exec consul-kv consul kv put first_value 100 \
&& docker exec consul-kv consul kv put second_value 200 \
&& docker exec consul-kv consul kv put third_value 300

cat <<EOF > tpl_1.ctmpl 
{{ key "first_value" }}
{{ key "second_value" }}
EOF

cat <<EOF > tpl_2.ctmpl 
{{ key "third_value" }}
{{ key "fourth_value" }}
EOF

docker run --rm \
 -e "CONSUL_HTTP_ADDR=consul-kv:8500" \
 -v $(pwd):/tpl \
 --network $DOCKER_NETWORK_NAME \
 --name consul-tpl \
 hashicorp/consul-template:0.33 \
 -log-level=debug \
 --wait=3s \
 -template '/tpl/tpl_1.ctmpl:/tpl/tpl_1.rendered:echo "$(date)" >> /tpl/tpl_1.deployed' \
 -template '/tpl/tpl_2.ctmpl:/tpl/tpl_2.rendered:echo "$(date)" >> /tpl/tpl_2.deployed'

References

Couldn't find similar issue.

[markotitel]

Can we exclude template which value is missing from Consul KV and reach global quiexcence for templates that have all keys in the KV?
Is it reasonable to render an empty file and set quiescence for empty template, then when KV is added consul-template will render the file?