Ruby thread creation issue in docker
Closed this issue · 6 comments
I have been using acmesmith for several years to update a wildcard certificate used by short-lived servers. As long as my cron-script has been running correctly, acmesmith has worked well.
Today we discovered that the certificate had expired. Looking at the log file, it seems that the autorenew function has not been able to complete since around 2023-10-04 (before that, it has been performing like clockwork).
I suspect it is related to whatever was changed in the 2.6.0 release, so I'm going to try using the docker container for the
This is the error log that happens when trying to run autorenew:
bundler: failed to load command: bin/acmesmith (bin/acmesmith)
/usr/lib/ruby/3.2.0/timeout.rb:101:in `initialize': can't create Thread: Operation not permitted (ThreadError)
from /usr/lib/ruby/3.2.0/timeout.rb:101:in `new'
from /usr/lib/ruby/3.2.0/timeout.rb:101:in `create_timeout_thread'
from /usr/lib/ruby/3.2.0/timeout.rb:134:in `block in ensure_timeout_thread_created'
from /usr/lib/ruby/3.2.0/timeout.rb:132:in `synchronize'
from /usr/lib/ruby/3.2.0/timeout.rb:132:in `ensure_timeout_thread_created'
from /usr/lib/ruby/3.2.0/timeout.rb:181:in `timeout'
from /usr/lib/ruby/3.2.0/net/http.rb:1269:in `connect'
from /usr/lib/ruby/3.2.0/net/http.rb:1248:in `do_start'
from /usr/lib/ruby/3.2.0/net/http.rb:1243:in `start'
from /usr/lib/ruby/3.2.0/delegate.rb:87:in `method_missing'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/seahorse/client/net_http/connection_pool.rb:307:in `start_session'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/seahorse/client/net_http/connection_pool.rb:100:in `session_for'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/seahorse/client/net_http/handler.rb:128:in `session'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/seahorse/client/net_http/handler.rb:76:in `transmit'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/seahorse/client/net_http/handler.rb:50:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/seahorse/client/plugins/content_length.rb:24:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/seahorse/client/plugins/request_callback.rb:118:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/plugins/s3_signer.rb:73:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/plugins/s3_host_id.rb:17:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/xml/error_handler.rb:10:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/sign.rb:49:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/transfer_encoding.rb:26:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/plugins/s3_signer.rb:48:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/plugins/redirects.rb:20:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/retry_errors.rb:360:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/user_agent.rb:37:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/http_checksum.rb:19:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:136:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/request_compression.rb:94:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/plugins/expect_100_continue.rb:23:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/plugins/bucket_name_restrictions.rb:21:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/rest/handler.rb:10:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/recursion_detection.rb:18:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/plugins/endpoints.rb:41:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/endpoint_discovery.rb:84:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/seahorse/client/plugins/endpoint.rb:47:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/plugins/url_encoded_keys.rb:43:in `manage_keys'
from /gems/ruby/3.2.0/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/plugins/url_encoded_keys.rb:35:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/seahorse/client/plugins/raise_response_errors.rb:16:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
from /gems/ruby/3.2.0/gems/aws-sdk-core-3.185.0/lib/seahorse/client/request.rb:72:in `send_request'
from /gems/ruby/3.2.0/gems/aws-sdk-s3-1.136.0/lib/aws-sdk-s3/client.rb:8682:in `list_objects'
from /app/lib/acmesmith/storages/s3.rb:118:in `list_certificates'
from /app/lib/acmesmith/client.rb:126:in `autorenew'
from /app/lib/acmesmith/command.rb:148:in `autorenew'
from /gems/ruby/3.2.0/gems/thor-1.2.2/lib/thor/command.rb:27:in `run'
from /gems/ruby/3.2.0/gems/thor-1.2.2/lib/thor/invocation.rb:127:in `invoke_command'
from /gems/ruby/3.2.0/gems/thor-1.2.2/lib/thor.rb:392:in `dispatch'
from /gems/ruby/3.2.0/gems/thor-1.2.2/lib/thor/base.rb:485:in `start'
from bin/acmesmith:4:in `<top (required)>'
from /var/lib/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli/exec.rb:58:in `load'
from /var/lib/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli/exec.rb:58:in `kernel_load'
from /var/lib/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli/exec.rb:23:in `run'
from /var/lib/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli.rb:492:in `exec'
from /var/lib/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
from /var/lib/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
from /var/lib/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
from /var/lib/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli.rb:34:in `dispatch'
from /var/lib/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
from /var/lib/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/cli.rb:28:in `start'
from /var/lib/gems/3.2.0/gems/bundler-2.4.13/exe/bundle:45:in `block in <top (required)>'
from /var/lib/gems/3.2.0/gems/bundler-2.4.13/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
from /var/lib/gems/3.2.0/gems/bundler-2.4.13/exe/bundle:33:in `<top (required)>'
from /usr/local/bin/bundle:25:in `load'
from /usr/local/bin/bundle:25:in `<main>'
I tried to use the docker image with the tag v2.5.0 (instead of latest), and everything worked as before, so it would seem that something in v2.6.0 is not working as expected (at least in the environment I'm using acmesmith).
As a side note: acmesmith is executed using docker via a bash script triggered by cron every 12 hours on an t3.small EC2 instance in AWS. The EC2 instance is running Debian Linux 9.13 (Stretch).
We rolled a new Ruby version (3.2.x) and base distro (Ubuntu 22.04) in our default Dockerfile, and I believe you have a problem with your container host environment. Debian Stretch sounds too outdated for me, could you try upgrading your container host to the latest distro first?
Maybe old Docker engine doesn't work well with newer glibc? moby/moby#42680
I cannot easily upgrade that server right now, but I tried using the latest docker image version and the same configuration on one of my recently upgraded servers (Debian 12) which worked perfectly 😅
For reference, the docker version on the old server is 19.03.15 (build 99e3ed8919) with containerd version 1.4.3.
On the newer server it's 24.0.6 (build ed223bc) with containerd version 1.6.24.
I don't know enough about the internals of docker/containerd, but I suppose it could very well be related to the issue you linked 🤷♂️
If anyone else with the same issue sees this, at least there is a workaround if an OS upgrade is not possible: use v2.5.0 😄
I looked into this a little bit more, and I was able to run v2.6.0 on the old server by adding --privileged=true to the docker run. It is not recommended by the docs, but at least it shows that there is some kind of low level permission issue related to threading. Perhaps it is something that has changed in more recent versions of the Docker Engine?
To repeat, I do not recommend to use both outdated distro on host and outdated base distro and runtime on the container image. You may build your own Docker image by trying other versions like Ubuntu 20.04. Refer to our Dockerfile and you may still be able to use https://github.com/sorah-rbpkg/dockerfiles for alternative distro versions.