Cron jobs failing with setuidgid: fatal: unable to run --: file does not exis
Closed this issue · 2 comments
Description of problem
Cron jobs are failing after upgrading from Ubuntu 20.04 to 22.04 and to Dokku 33.1.
Jobs are in the Dokku crontab file.
Steps to reproduce
Add something to cron with app.json file.
dokku report $APP_NAME
-----> uname: Linux dokku 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
-----> memory:
total used free shared buff/cache available
Mem: 3912 1713 491 195 1706 1711
Swap: 2047 689 1358
-----> docker version:
Client: Docker Engine - Community
Version: 25.0.1
API version: 1.44
Go version: go1.21.6
Git commit: 29cf629
Built: Tue Jan 23 23:09:25 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 25.0.1
API version: 1.44 (minimum version 1.24)
Go version: go1.21.6
Git commit: 71fa3ab
Built: Tue Jan 23 23:09:25 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.27
GitCommit: a1496014c916f9e62104b33d1bb5bd03b0858e59
runc:
Version: 1.1.11
GitCommit: v1.1.11-0-g4bccb38
docker-init:
Version: 0.19.0
GitCommit: de40ad0
-----> docker daemon info:
Client: Docker Engine - Community
Version: 25.0.1
Context: default
Debug Mode: true
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.12.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.24.2
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 20
Running: 20
Paused: 0
Stopped: 0
Images: 56
Server Version: 25.0.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
runc version: v1.1.11-0-g4bccb38
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-91-generic
Operating System: Ubuntu 22.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.82GiB
Name: dokku
ID: e0f15899-29a0-4d65-bd98-3c6a8d03c7e3
Docker Root Dir: /var/lib/docker
Debug Mode: false
File Descriptors: 146
Goroutines: 129
System Time: 2024-01-29T16:52:38.308877862Z
EventsListeners: 1
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
-----> herokuish version:
herokuish: v0.7.2
buildpacks:
heroku-buildpack-multi v1.2.0
heroku-buildpack-ruby v259
heroku-buildpack-nodejs v224
heroku-buildpack-clojure v91
heroku-buildpack-python v237
heroku-buildpack-java v73
heroku-buildpack-gradle v39
heroku-buildpack-scala v96
heroku-buildpack-play v26
heroku-buildpack-php v239
heroku-buildpack-go v181
heroku-buildpack-nginx v24
buildpack-null v3
-----> dokku version: dokku version 0.33.1
-----> dokku-event-listener version: 0.15.0build+5268732
-----> dokku-update version: dokku-update 0.7.2
-----> docker-container-healthchecker version: 0.7.2
-----> docker-image-labeler version: 0.6.1build+c6e15a9
-----> git version: git version 2.34.1
-----> lambda-builder version: 0.5.0
-----> netrc version: 0.7.1build+677c033
! pack binary is not available
-----> plugn version: plugn: 0.13.0build+fd5297a
-----> sigil version: 0.10.1build+e443be0
-----> sshcommand version: sshcommand 0.17.1
-----> dokku plugins:
00_dokku-standard 0.33.1 enabled dokku core standard plugin
20_events 0.33.1 enabled dokku core events logging plugin
app-json 0.33.1 enabled dokku core app-json plugin
apps 0.33.1 enabled dokku core apps plugin
builder 0.33.1 enabled dokku core builder plugin
builder-dockerfile 0.33.1 enabled dokku core builder-dockerfile plugin
builder-herokuish 0.33.1 enabled dokku core builder-herokuish plugin
builder-lambda 0.33.1 enabled dokku core builder-lambda plugin
builder-nixpacks 0.33.1 enabled dokku core builder-nixpacks plugin
builder-null 0.33.1 enabled dokku core builder-null plugin
builder-pack 0.33.1 enabled dokku core builder-pack plugin
buildpacks 0.33.1 enabled dokku core buildpacks plugin
caddy-vhosts 0.33.1 enabled dokku core caddy-vhosts plugin
certs 0.33.1 enabled dokku core certificate management plugin
checks 0.33.1 enabled dokku core checks plugin
common 0.33.1 enabled dokku core common plugin
config 0.33.1 enabled dokku core config plugin
cron 0.33.1 enabled dokku core cron plugin
docker-options 0.33.1 enabled dokku core docker-options plugin
domains 0.33.1 enabled dokku core domains plugin
enter 0.33.1 enabled dokku core enter plugin
git 0.33.1 enabled dokku core git plugin
haproxy-vhosts 0.33.1 enabled dokku core haproxy-vhosts plugin
letsencrypt 0.20.3 enabled Automated installation of let's encrypt TLS certificates
litestream 0.2.0 enabled Litestream plugin to replicate SQLite DB files from Dokku
logs 0.33.1 enabled dokku core logs plugin
network 0.33.1 enabled dokku core network plugin
nginx-vhosts 0.33.1 enabled dokku core nginx-vhosts plugin
openresty-vhosts 0.33.1 enabled dokku core openresty-vhosts plugin
plugin 0.33.1 enabled dokku core plugin plugin
ports 0.33.1 enabled dokku core ports plugin
postgres 1.11.6 enabled dokku postgres service plugin
proxy 0.33.1 enabled dokku core proxy plugin
ps 0.33.1 enabled dokku core ps plugin
redis 1.34.0 enabled dokku redis service plugin
registry 0.33.1 enabled dokku core registry plugin
repo 0.33.1 enabled dokku core repo plugin
resource 0.33.1 enabled dokku core resource plugin
run 0.33.1 enabled dokku core run plugin
scheduler 0.33.1 enabled dokku core scheduler plugin
scheduler-docker-local 0.33.1 enabled dokku core scheduler-docker-local plugin
scheduler-k3s 0.33.1 enabled dokku core scheduler-k3s plugin
scheduler-null 0.33.1 enabled dokku core scheduler-null plugin
shell 0.33.1 enabled dokku core shell plugin
ssh-keys 0.33.1 enabled dokku core ssh-keys plugin
storage 0.33.1 enabled dokku core storage plugin
trace 0.33.1 enabled dokku core trace plugin
traefik-vhosts 0.33.1 enabled dokku core traefik-vhosts plugin
=====> commandwp app-json information
App json computed selected: app.json
App json global selected: app.json
App json selected:
=====> commandwp app information
App created at: 1688073339
App deploy source: git-push
App deploy source metadata: 38056735d1ea4d9a6333d3755d2790d382aa925f
App dir: /home/dokku/commandwp
App locked: false
=====> commandwp builder information
Builder build dir:
Builder computed build dir:
Builder computed selected:
Builder global build dir:
Builder global selected:
Builder selected:
=====> commandwp builder-dockerfile information
Builder dockerfile computed dockerfile path: Dockerfile
Builder dockerfile global dockerfile path: Dockerfile
Builder dockerfile dockerfile path:
=====> commandwp builder-herokuish information
Builder herokuish computed allowed: true
Builder herokuish global allowed: true
Builder herokuish allowed:
=====> commandwp builder-lambda information
Builder lambda computed lambdayml path: lambda.yml
Builder lambda global lambdayml path: lambda.yml
Builder lambda lambdayml path:
=====> commandwp builder-nixpacks information
Builder nixpacks computed nixpackstoml path: nixpacks.toml
Builder nixpacks global nixpackstoml path: nixpacks.toml
Builder nixpacks nixpackstoml path:
Builder nixpacks computed no cache: false
Builder nixpacks global no cache: false
Builder nixpacks no cache:
=====> commandwp builder-pack information
Builder pack computed projecttoml path: project.toml
Builder pack global projecttoml path: project.toml
Builder pack projecttoml path:
=====> commandwp buildpacks information
Buildpacks computed stack: gliderlabs/herokuish:latest-22
Buildpacks global stack:
Buildpacks list: https://github.com/heroku/heroku-buildpack-apt,https://github.com/heroku/heroku-buildpack-ruby.git
Buildpacks stack:
=====> commandwp caddy information
Caddy image: lucaslorentz/caddy-docker-proxy:2.8
Caddy letsencrypt email:
Caddy letsencrypt server: https://acme-v02.api.letsencrypt.org/directory
Caddy log level: ERROR
Caddy polling interval: 5s
Caddy tls internal: false
=====> commandwp ssl information
Ssl dir: /home/dokku/commandwp/tls
Ssl enabled: true
Ssl hostnames: app.commandwp.com commandwp.dokku.red27.net
Ssl expires at: Apr 28 14:13:18 2024 GMT
Ssl issuer: C = US, O = Let's Encrypt, CN = R3
Ssl starts at: Jan 29 14:13:19 2024 GMT
Ssl subject: subject=CN = app.commandwp.com
Ssl verified: self signed
=====> commandwp checks information
Checks disabled list: none
Checks skipped list: none
Checks computed wait to retire: 60
Checks global wait to retire: 60
Checks wait to retire:
=====> commandwp docker options information
Docker options build: --link dokku.postgres.commandwp_production:dokku-postgres-commandwp-production --link dokku.redis.commandwp_redis:dokku-redis-commandwp-redis
Docker options deploy: --link dokku.postgres.commandwp_production:dokku-postgres-commandwp-production --link dokku.redis.commandwp_redis:dokku-redis-commandwp-redis --restart=on-failure:10
Docker options run: --link dokku.postgres.commandwp_production:dokku-postgres-commandwp-production --link dokku.redis.commandwp_redis:dokku-redis-commandwp-redis
=====> commandwp domains information
Domains app enabled: true
Domains app vhosts: app.commandwp.com commandwp.dokku.red27.net
Domains global enabled: true
Domains global vhosts: dokku.red27.net
=====> commandwp git information
Git deploy branch: master
Git global deploy branch: master
Git keep git dir: false
Git rev env var: GIT_REV
Git sha: 3805673
Git source image:
Git last updated at: 1706541142
=====> commandwp haproxy information
Haproxy image: byjg/easy-haproxy:4.3.0
Haproxy letsencrypt email:
Haproxy letsencrypt server: https://acme-v02.api.letsencrypt.org/directory
Haproxy log level: ERROR
=====> commandwp letsencrypt information
Letsencrypt active: true
Letsencrypt autorenew: true
Letsencrypt computed dns provider:
Letsencrypt global dns provider:
Letsencrypt dns provider:
Letsencrypt computed email: dusty@red27.net
Letsencrypt global email:
Letsencrypt email: dusty@red27.net
Letsencrypt expiration: 1714313598
Letsencrypt computed graceperiod: 2592000
Letsencrypt global graceperiod:
Letsencrypt graceperiod:
Letsencrypt computed lego docker args:
Letsencrypt global lego docker args:
Letsencrypt lego docker args:
Letsencrypt computed server: https://acme-v02.api.letsencrypt.org/directory
Letsencrypt global server:
Letsencrypt server:
=====> commandwp logs information
Logs computed max size: 10m
Logs global max size: 10m
Logs global vector sink:
Logs max size:
Logs vector sink:
=====> commandwp network information
Network attach post create:
Network attach post deploy:
Network bind all interfaces: false
Network computed attach post create:
Network computed attach post deploy:
Network computed bind all interfaces: false
Network computed initial network:
Network computed tld:
Network global attach post create:
Network global attach post deploy:
Network global bind all interfaces: false
Network global initial network:
Network global tld:
Network initial network:
Network static web listener:
Network tld:
Network web listeners: 172.17.0.13:5000
=====> commandwp nginx information
Nginx access log format:
Nginx access log path: /var/log/nginx/commandwp-access.log
Nginx bind address ipv4:
Nginx bind address ipv6: ::
Nginx client max body size:
Nginx disable custom config: false
Nginx error log path: /var/log/nginx/commandwp-error.log
Nginx global hsts: true
Nginx computed hsts: true
Nginx hsts:
Nginx hsts include subdomains: true
Nginx hsts max age: 15724800
Nginx hsts preload: false
Nginx computed nginx conf sigil path: nginx.conf.sigil
Nginx global nginx conf sigil path: nginx.conf.sigil
Nginx nginx conf sigil path:
Nginx proxy buffer size: 4096
Nginx proxy buffering: on
Nginx proxy buffers: 8 4096
Nginx proxy busy buffers size: 8192
Nginx proxy read timeout: 60s
Nginx last visited at: 1706547150
Nginx x forwarded for value: $remote_addr
Nginx x forwarded port value: $server_port
Nginx x forwarded proto value: $scheme
Nginx x forwarded ssl:
=====> commandwp openresty information
Openresty access log format:
Openresty access log path: /var/log/nginx/commandwp-access.log
Openresty bind address ipv4:
Openresty bind address ipv6: ::
Openresty client max body size:
Openresty error log path: /var/log/nginx/commandwp-error.log
Openresty global hsts: true
Openresty computed hsts: true
Openresty hsts:
Openresty hsts include subdomains: true
Openresty hsts max age: 15724800
Openresty hsts preload: false
Openresty image: dokku/openresty-docker-proxy:0.6.0
Openresty letsencrypt email:
Openresty letsencrypt server: https://acme-v02.api.letsencrypt.org/directory
Openresty proxy buffer size: 4096
Openresty proxy buffering: on
Openresty proxy buffers: 8 4096
Openresty proxy busy buffers size: 8192
Openresty proxy read timeout: 60s
Openresty x forwarded for value: $remote_addr
Openresty x forwarded port value: $server_port
Openresty x forwarded proto value: $scheme
Openresty x forwarded ssl:
=====> commandwp ports information
Ports map: http:80:5000 https:443:5000
Ports map detected: http:80:5000 https:443:5000
=====> commandwp proxy information
Proxy computed type: nginx
Proxy enabled: true
Proxy global type: nginx
Proxy type:
=====> commandwp ps information
Deployed: true
Processes: 2
Ps can scale: true
Ps computed procfile path: Procfile
Ps global procfile path: Procfile
Ps procfile path:
Ps restart policy: on-failure:10
Restore: true
Running: true
Status web 1: running (CID: c122a0c5103)
Status worker 1: running (CID: db18ea8cb1c)
=====> commandwp registry information
Registry computed image repo: dokku/commandwp
Registry computed push on release: false
Registry computed server:
Registry global image repo template:
Registry global push on release:
Registry global server:
Registry image repo:
Registry push on release:
Registry server:
Registry tag version:
=====> commandwp resource information
=====> commandwp scheduler information
Scheduler computed selected: docker-local
Scheduler global selected: docker-local
Scheduler selected:
=====> commandwp scheduler-docker-local information
Scheduler docker local disable chown:
Scheduler docker local init process: true
Scheduler docker local parallel schedule count:
=====> commandwp scheduler-k3s information
Scheduler k3s computed deploy timeout: 300s
Scheduler k3s computed image pull secrets:
Scheduler k3s computed letsencrypt server: prod
Scheduler k3s computed namespace: default
Scheduler k3s computed rollback on failure: false
Scheduler k3s deploy timeout:
Scheduler k3s global deploy timeout: 300s
Scheduler k3s global image pull secrets:
Scheduler k3s global letsencrypt email prod:
Scheduler k3s global letsencrypt email stag:
Scheduler k3s global letsencrypt server: prod
Scheduler k3s global namespace: default
Scheduler k3s global network interface: eth0
Scheduler k3s global rollback on failure: false
Scheduler k3s image pull secrets:
Scheduler k3s letsencrypt server:
Scheduler k3s namespace:
Scheduler k3s rollback on failure:
=====> commandwp storage information
Storage build mounts:
Storage deploy mounts:
Storage run mounts:
=====> commandwp traefik information
Traefik api enabled: false
Traefik api vhost: traefik.dokku.me
Traefik basic auth password:
Traefik basic auth username:
Traefik dashboard enabled: false
Traefik image: traefik:v2.10
Traefik letsencrypt email:
Traefik letsencrypt server: https://acme-v02.api.letsencrypt.org/directory
Traefik log level: ERROR
Traefik priority:
Additional information
root@dokku:~# cat /var/spool/cron/crontabs/dokku
DO NOT EDIT THIS FILE - edit the master and reinstall.
(/tmp/crontab.0YJwd4/crontab installed on Mon Jan 29 15:17:47 2024)
(Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $ )
PATH=/usr/local/bin:/usr/bin:/bin
SHELL=/bin/bash
@daily dokku letsencrypt:auto-renew &>> /var/log/dokku/letsencrypt.log
30 6 * * 0 dokku run --cron-id kk9rs3zvih5n09o7nc0sjvjxtydsht7t4ixa6itvp1ufmruzgzjagsufcleatlojvm203yisrrmkkkk08kunk commandwp -- /app/bin/rake reports:security
45 6 * * 0 dokku run --cron-id 223zia637rqzma0d334ohw933sdkzuwfuxsa9qbfmzvzpm9x9hrxtux50md1xyp0vfvzvx6990sv3ncxc commandwp -- /app/bin/rake reports:basic
0 7 * * * dokku run --cron-id 223zia637rqzma0d334ohw933sdkzuwfuxsa9qbfn0vghciyfqcmhs00wgj12u192juavgfq7aw2a73my commandwp -- /app/bin/rake uptime:refresh
5 7 * * * dokku run --cron-id svtz0tijh43kpnt4df25ftk239fkip4zrxqr1iav0bgg874av5e3rvwpl54awfw17vh26u7r8mob4481yck9snqi commandwp -- /app/bin/rake wordpress:configure
15 7 * * * dokku run --cron-id en0ci0bav9bxaf6l1y9a79cm2xslazqos095175b7ut4pebrlj38j59lrf8jsv32jtn449929u82p3o0mi commandwp -- /app/bin/rake wordpress:sync
45 7 * * * dokku run --cron-id kk9rs3zvih5n09o7nc0sjvjxtydsht7t4ixa6itvpi52eu3l8gqo4rqrmc5s8uf9yezmpnuee1d9kbvaaavje commandwp -- /app/bin/rake wordpress:backup
45 9 * * * dokku run --cron-id 7f3lgemyxgn8aiguq5c4yvmzdpyqff13lipjsdmgtmttq8csoma1frse8980lz78kw4ziwsmw3l6 commandwp -- /app/bin/rake cleanup:s3
0 9 * * * dokku run --cron-id 9nalqy318dkw7kg9ry64y3u12ua6n1fc7gs1zjisxy8n9li4sn3qny242jwiei cloudsh -- rake csh:index:all
0 7 * * * dokku run --cron-id 11k18dv9uktejl86yhiaj7gah6ur9sk4cdifmm1cet5girvhel8ly7n32gvs8ocadvlf2nrhlwq cloudsh -- rake csh:admin:clean_users
0 8 * * * dokku run --cron-id 6v8s2hg0f50zl4yn9t2m0ijt78bqtgoiqq89mdqqc8prxuytpu8yeb4wkkq cloudsh -- rake stripe:sync
Output of failing commands after running: dokku trace:off
No response
Output of failing commands after running: dokku trace:on
No response
Mind checking to see if updating the crontab to remove the -- manually makes things work? You can do sudo service cron restart afterwards to make it take effect.
If that fixes it, the above patch should work.
Removing the -- does fix things.
Thanks @josegonzalez!