Caddy is not reading my labels
sowinski opened this issue · 4 comments
sowinski commented
Hi,
this is my docker-compose for caddy:
version: "3.7"
services:
caddy:
image: lucaslorentz/caddy-docker-proxy:2.8.9-alpine
ports:
- 80:80
- 443:443
environment:
- CADDY_INGRESS_NETWORKS=caddy
networks:
- caddy
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- caddy_data:/data
restart: unless-stopped
networks:
caddy:
external: true
volumes:
caddy_data: {}
And this is my docker-compose for my django project.
version: '3.8'
services:
django:
image: django-backend:latest
ports:
- "8000:8000"
command: gunicorn myapp.wsgi --bind 0.0.0.0:8000
env_file:
- ../django.env
environment:
CELERY_BROKER_URL: ${CELERY_BROKER_URL}
volumes:
- logvolume:/code/logs
networks:
- caddy
labels:
caddy: www.example.com
caddy.reverse_proxy: "{{upstreams 8000}}"
volumes:
logvolume:
networks:
caddy:
external: true
The first time i used the port 9999 and changed it later to 8000.
I restarted several times caddy and also my django project.
I can locally connect to the django project and it works on port 8000.
If I connect externally from www.example.com i can see in the caddy logs that caddy is still trying to forward it to port 9999.
I do not undestand why it is not updating.
Do I have to implement a healthcheck or something else to make caddy update this to port 8000?
francislavoie commented
Please show your logs.
sowinski commented
sudo docker-compose up --build
[+] Running 1/0
✔ Container caddy_caddy_1 Created 0.0s
Attaching to caddy_1
caddy_1 | {"level":"info","ts":1700776180.582202,"logger":"docker-proxy","msg":"Running caddy proxy server"}
caddy_1 | {"level":"info","ts":1700776180.5840368,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy_1 | {"level":"info","ts":1700776180.5844781,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1 | {"level":"info","ts":1700776180.5844936,"logger":"docker-proxy","msg":"Running caddy proxy controller"}
caddy_1 | {"level":"info","ts":1700776180.5854888,"logger":"docker-proxy","msg":"Start","CaddyfilePath":"","LabelPrefix":"caddy","PollingInterval":30,"ProxyServiceTasks":true,"ProcessCaddyfile":true,"ScanStoppedContainers":true,"IngressNetworks":"[caddy]","DockerSockets":[""],"DockerCertsPath":[""],"DockerAPIsVersion":[""]}
caddy_1 | {"level":"info","ts":1700776180.5866935,"logger":"docker-proxy","msg":"Connecting to docker events","DockerSocket":""}
caddy_1 | {"level":"info","ts":1700776180.588175,"logger":"docker-proxy","msg":"IngressNetworksMap","ingres":"map[a48a2cad8e93313766a61789781a28adc75c571c86cd47b4e09aa713d1c3ae9c:true caddy:true]"}
caddy_1 | {"level":"info","ts":1700776180.595984,"logger":"docker-proxy","msg":"Swarm is available","new":false}
caddy_1 | {"level":"info","ts":1700776180.60986,"logger":"docker-proxy","msg":"New Caddyfile","caddyfile":"www.example.com {\n\treverse_proxy 172.29.0.3:8000 172.29.0.4:9999 :9999 :9999 :80\n}\n"}
caddy_1 | {"level":"info","ts":1700776180.6102848,"logger":"docker-proxy","msg":"New Config JSON","json":"{\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":443\"],\"routes\":[{\"match\":[{\"host\":[\"www.example.com\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"172.29.0.3:8000\"},{\"dial\":\"172.29.0.4:9999\"},{\"dial\":\":9999\"},{\"dial\":\":9999\"},{\"dial\":\":80\"}]}]}]}],\"terminal\":true}]}}}}}"}
caddy_1 | {"level":"info","ts":1700776180.6103382,"logger":"docker-proxy","msg":"Sending configuration to","server":"localhost"}
caddy_1 | {"level":"info","ts":1700776180.6109803,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"42482","headers":{"Accept-Encoding":["gzip"],"Content-Length":["367"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy_1 | {"level":"info","ts":1700776180.611386,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
caddy_1 | {"level":"info","ts":1700776180.6114962,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1 | {"level":"info","ts":1700776180.6115198,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1 | {"level":"info","ts":1700776180.611741,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy_1 | {"level":"info","ts":1700776180.6118507,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
caddy_1 | {"level":"info","ts":1700776180.611999,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy_1 | {"level":"info","ts":1700776180.612044,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy_1 | {"level":"info","ts":1700776180.6120574,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.example.com"]}
caddy_1 | {"level":"info","ts":1700776180.6131058,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1 | {"level":"info","ts":1700776180.6131315,"logger":"admin.api","msg":"load complete"}
caddy_1 | {"level":"info","ts":1700776180.6133792,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy_1 | {"level":"info","ts":1700776180.613424,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0006bc600"}
caddy_1 | {"level":"info","ts":1700776180.6136477,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddy_1 | {"level":"info","ts":1700776180.614009,"logger":"tls","msg":"finished cleaning storage units"}
caddy_1 | {"level":"info","ts":1700776180.6143267,"logger":"docker-proxy","msg":"Successfully configured","server":"localhost"}
caddy_1 | {"level":"info","ts":1700776189.0448728,"logger":"docker-proxy","msg":"New Caddyfile","caddyfile":"www.example.com {\n\treverse_proxy :8000 172.29.0.4:9999 :9999 :9999 :80\n}\n"}
caddy_1 | {"level":"info","ts":1700776189.045548,"logger":"docker-proxy","msg":"New Config JSON","json":"{\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":443\"],\"routes\":[{\"match\":[{\"host\":[\"www.example.com\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\":8000\"},{\"dial\":\"172.29.0.4:9999\"},{\"dial\":\":9999\"},{\"dial\":\":9999\"},{\"dial\":\":80\"}]}]}]}],\"terminal\":true}]}}}}}"}
caddy_1 | {"level":"info","ts":1700776189.0455964,"logger":"docker-proxy","msg":"Sending configuration to","server":"localhost"}
caddy_1 | {"level":"info","ts":1700776189.0471437,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"36668","headers":{"Accept-Encoding":["gzip"],"Content-Length":["357"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy_1 | {"level":"info","ts":1700776189.04832,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
caddy_1 | {"level":"info","ts":1700776189.048497,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1 | {"level":"info","ts":1700776189.0485358,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1 | {"level":"info","ts":1700776189.0490217,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy_1 | {"level":"info","ts":1700776189.0491111,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy_1 | {"level":"info","ts":1700776189.0492048,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy_1 | {"level":"info","ts":1700776189.049262,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.example.com"]}
caddy_1 | {"level":"info","ts":1700776189.0492947,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy_1 | {"level":"info","ts":1700776189.0501091,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1 | {"level":"info","ts":1700776189.05016,"logger":"admin.api","msg":"load complete"}
caddy_1 | {"level":"info","ts":1700776189.050379,"logger":"docker-proxy","msg":"Successfully configured","server":"localhost"}
caddy_1 | {"level":"info","ts":1700776189.0505602,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy_1 | {"level":"info","ts":1700776192.842112,"logger":"docker-proxy","msg":"New Caddyfile","caddyfile":"www.example.com {\n\treverse_proxy 172.29.0.3:8000 172.29.0.4:9999 :9999 :9999 :80\n}\n"}
caddy_1 | {"level":"info","ts":1700776192.8425071,"logger":"docker-proxy","msg":"New Config JSON","json":"{\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":443\"],\"routes\":[{\"match\":[{\"host\":[\"www.example.com\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"172.29.0.3:8000\"},{\"dial\":\"172.29.0.4:9999\"},{\"dial\":\":9999\"},{\"dial\":\":9999\"},{\"dial\":\":80\"}]}]}]}],\"terminal\":true}]}}}}}"}
caddy_1 | {"level":"info","ts":1700776192.8425362,"logger":"docker-proxy","msg":"Sending configuration to","server":"localhost"}
caddy_1 | {"level":"info","ts":1700776192.8434806,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"36680","headers":{"Accept-Encoding":["gzip"],"Content-Length":["367"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy_1 | {"level":"info","ts":1700776192.8449876,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
caddy_1 | {"level":"info","ts":1700776192.8451831,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1 | {"level":"info","ts":1700776192.845202,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1 | {"level":"info","ts":1700776192.8456044,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy_1 | {"level":"info","ts":1700776192.8456333,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy_1 | {"level":"info","ts":1700776192.8456771,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy_1 | {"level":"info","ts":1700776192.8456895,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.example.com"]}
caddy_1 | {"level":"info","ts":1700776192.8457038,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy_1 | {"level":"info","ts":1700776192.846989,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1 | {"level":"info","ts":1700776192.847001,"logger":"admin.api","msg":"load complete"}
caddy_1 | {"level":"info","ts":1700776192.847502,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy_1 | {"level":"info","ts":1700776192.8476045,"logger":"docker-proxy","msg":"Successfully configured","server":"localhost"}
caddy_1 | {"level":"error","ts":1700776220.799892,"logger":"http.log.error","msg":"dial tcp :9999: connect: connection refused","request":{"remote_ip":"172.29.0.1","remote_port":"54236","client_ip":"172.29.0.1","proto":"HTTP/2.0","method":"GET","host":"www.example.com","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.115 Safari/537.36"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\""],"Sec-Ch-Ua-Platform":["\"Linux\""],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Cache-Control":["max-age=0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"www.example.com"}},"duration":0.000555354,"status":502,"err_id":"ebd259cur","err_trace":"reverseproxy.statusError (reverseproxy.go:1265)"}
caddy_1 | {"level":"info","ts":1700776305.57138,"logger":"docker-proxy","msg":"New Caddyfile","caddyfile":"www.example.com {\n\treverse_proxy :8000 172.29.0.4:9999 :9999 :9999 :80\n}\n"}
caddy_1 | {"level":"info","ts":1700776305.5729163,"logger":"docker-proxy","msg":"New Config JSON","json":"{\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":443\"],\"routes\":[{\"match\":[{\"host\":[\"www.example.com\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\":8000\"},{\"dial\":\"172.29.0.4:9999\"},{\"dial\":\":9999\"},{\"dial\":\":9999\"},{\"dial\":\":80\"}]}]}]}],\"terminal\":true}]}}}}}"}
caddy_1 | {"level":"info","ts":1700776305.5729902,"logger":"docker-proxy","msg":"Sending configuration to","server":"localhost"}
caddy_1 | {"level":"info","ts":1700776305.5753102,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"34442","headers":{"Accept-Encoding":["gzip"],"Content-Length":["357"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy_1 | {"level":"info","ts":1700776305.576389,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy_1 | {"level":"info","ts":1700776305.5765822,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1 | {"level":"info","ts":1700776305.5766027,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1 | {"level":"info","ts":1700776305.577017,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy_1 | {"level":"info","ts":1700776305.5770495,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy_1 | {"level":"info","ts":1700776305.577086,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy_1 | {"level":"info","ts":1700776305.577091,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.example.com"]}
caddy_1 | {"level":"info","ts":1700776305.5771034,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy_1 | {"level":"info","ts":1700776305.577657,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1 | {"level":"info","ts":1700776305.5776682,"logger":"admin.api","msg":"load complete"}
caddy_1 | {"level":"info","ts":1700776305.577816,"logger":"docker-proxy","msg":"Successfully configured","server":"localhost"}
caddy_1 | {"level":"info","ts":1700776305.5779428,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
caddy_1 | {"level":"info","ts":1700776309.1495879,"logger":"docker-proxy","msg":"New Caddyfile","caddyfile":"www.example.com {\n\treverse_proxy 172.29.0.4:9999 :9999 :9999 :80\n}\n"}
caddy_1 | {"level":"info","ts":1700776309.149949,"logger":"docker-proxy","msg":"New Config JSON","json":"{\"apps\":{\"http\":{\"servers\":{\"srv0\":{\"listen\":[\":443\"],\"routes\":[{\"match\":[{\"host\":[\"www.example.com\"]}],\"handle\":[{\"handler\":\"subroute\",\"routes\":[{\"handle\":[{\"handler\":\"reverse_proxy\",\"upstreams\":[{\"dial\":\"172.29.0.4:9999\"},{\"dial\":\":9999\"},{\"dial\":\":9999\"},{\"dial\":\":80\"}]}]}]}],\"terminal\":true}]}}}}}"}
caddy_1 | {"level":"info","ts":1700776309.149974,"logger":"docker-proxy","msg":"Sending configuration to","server":"localhost"}
caddy_1 | {"level":"info","ts":1700776309.1504982,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"49218","headers":{"Accept-Encoding":["gzip"],"Content-Length":["340"],"Content-Type":["application/json"],"User-Agent":["Go-http-client/1.1"]}}
caddy_1 | {"level":"info","ts":1700776309.1510074,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy_1 | {"level":"info","ts":1700776309.1511235,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy_1 | {"level":"info","ts":1700776309.1511562,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy_1 | {"level":"info","ts":1700776309.1513488,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy_1 | {"level":"info","ts":1700776309.151398,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy_1 | {"level":"info","ts":1700776309.151426,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy_1 | {"level":"info","ts":1700776309.151437,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.example.com"]}
caddy_1 | {"level":"info","ts":1700776309.1514816,"logger":"http","msg":"servers shutting down with eternal grace period"}
caddy_1 | {"level":"info","ts":1700776309.1516933,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy_1 | {"level":"info","ts":1700776309.1517143,"logger":"admin.api","msg":"load complete"}
caddy_1 | {"level":"info","ts":1700776309.1519494,"logger":"docker-proxy","msg":"Successfully configured","server":"localhost"}
caddy_1 | {"level":"info","ts":1700776309.1529338,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
In the logs you can still see port 9999
francislavoie commented
caddy_1 | {"level":"info","ts":1700776309.1495879,"logger":"docker-proxy","msg":"New Caddyfile","caddyfile":"www.example.com {\n\treverse_proxy 172.29.0.4:9999 :9999 :9999 :80\n}\n"}
This tells me that you must still have some containers that were stopped but not removed from, or some other dangling state of somekind.
Please shut down all containers using docker compose down for each project, run docker system prune -f, then bring your containers back up.
sowinski commented
You are right. Thank you!