Bug: ryuk exited instances leaking
shoffmeister opened this issue · 4 comments
shoffmeister commented
Describe the bug
With the new ryuk container usage in v4.1.1, dependent containers get cleaned up nicely, but the ryuk container itself remains, in exited state. The ryuk container itself should also be cleaned up.
docker ps --all will show a great many of these containers, see the very end of the bug report for some testcontainers-based pytest integration test that was running in a loop.
While docker rm -f $(docker ps --all --quiet --filter="name=testcontainers-ryuk*") will clean this up, it is still annoying to have to manage this.
To Reproduce
Any usage of testcontainers should exhibit this behaviour.
Locally, I am running this with a specialized Dockercontainer implementation which drives the official Apache Kafka 3.7.0 container image in KRaft mode (which obviates the need for Zookeeper dancing and hence cycles much much faster).
Runtime environment
Provide a summary of your runtime environment. Which operating system, python version, and docker version are you using? What is the version of testcontainers-python you are using? You can run the following commands to get the relevant information.
# Get the operating system information (on a unix os).
$ uname -a
Linux arch 6.8.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 16 Mar 2024 17:15:35 +0000 x86_64 GNU/Linux
# Get the python version.
$ python --version
Python 3.11.8
# Get the docker version and other docker information.
$ docker info
❯ docker info
Client:
Version: 25.0.4
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: 0.13.1
Path: /usr/lib/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: 2.24.7
Path: /usr/lib/docker/cli-plugins/docker-compose
Server:
Containers: 50
Running: 0
Paused: 0
Stopped: 50
Images: 34
Server Version: 25.0.4
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: true
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: dcf2847247e18caba8dce86522029642f60fe96b.m
runc version:
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.1-arch1-1
Operating System: EndeavourOS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 31.31GiB
Name: arch
ID: 9b2395c3-f6f6-4d7d-b25b-4b0d963287ae
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
# Get all python packages.
$ pip freeze
asttokens==2.4.1
backoff==2.2.1
busypie==0.5.1
certifi==2024.2.2
charset-normalizer==3.3.2
comm==0.2.2
confluent-kafka==2.3.0
confluent-kafka-stubs @ git+https://gitlab.com/shoffmeister/confluent-kafka-stubs.git@29b72e31099a89609107c541ef6f9dfef3185164
debugpy==1.8.1
decorator==5.1.1
docker==7.0.0
duckdb==0.10.1
duckdb_engine==0.11.2
executing==2.0.1
greenlet==3.0.3
idna==3.6
iniconfig==2.0.0
ipykernel==6.29.3
ipython==8.22.2
ipython-genutils==0.2.0
ipython-sql==0.5.0
jedi==0.19.1
Jinja2==3.1.3
jupysql==0.10.10
jupysql-plugin==0.4.2
jupyter_client==8.6.1
jupyter_core==5.7.2
kafka-python==2.0.2
MarkupSafe==2.1.5
matplotlib-inline==0.1.6
monotonic==1.6
mypy==1.9.0
mypy-extensions==1.0.0
nest-asyncio==1.6.0
packaging==24.0
parso==0.8.3
pexpect==4.9.0
platformdirs==4.2.0
ploomber-core==0.2.25
ploomber-extension==0.1.0
pluggy==1.4.0
posthog==3.5.0
prettytable==3.10.0
prompt-toolkit==3.0.43
psutil==5.9.8
ptyprocess==0.7.0
pure-eval==0.2.2
Pygments==2.17.2
pytest==8.1.1
pytest-dependency==0.6.0
python-dateutil==2.9.0.post0
PyYAML==6.0.1
pyzmq==25.1.2
requests==2.31.0
ruff==0.3.2
six==1.16.0
SQLAlchemy==2.0.28
sqlglot==23.0.3
sqlparse==0.4.4
stack-data==0.6.3
testcontainers==4.1.1
tornado==6.4
traitlets==5.14.2
typing_extensions==4.10.0
urllib3==2.2.1
wcwidth==0.2.13
wrapt==1.16.0
1feb2e86bec7 testcontainers/ryuk:0.5.1 "/bin/ryuk" 36 seconds ago Exited (0) 15 seconds ago testcontainers-ryuk-e881adfb-5f0e-4f37-980d-cc96cb98e77b
baa9afb86caa testcontainers/ryuk:0.5.1 "/bin/ryuk" 51 seconds ago Exited (0) 26 seconds ago testcontainers-ryuk-753d4e33-b754-40ea-803a-44d5e3754f81
c3b72be91821 testcontainers/ryuk:0.5.1 "/bin/ryuk" 58 seconds ago Exited (0) 41 seconds ago testcontainers-ryuk-a9a84681-93ef-4a6a-a0ec-43bc4227566e
981a0d4be1c1 testcontainers/ryuk:0.5.1 "/bin/ryuk" About a minute ago Exited (0) 48 seconds ago testcontainers-ryuk-53d2d06e-48c1-4f52-9045-5c568c67546c
031887f37bf5 testcontainers/ryuk:0.5.1 "/bin/ryuk" About a minute ago Exited (0) 50 seconds ago testcontainers-ryuk-ea29c45b-a187-4e8c-9a98-a788a7b71540
571fa2953a96 testcontainers/ryuk:0.5.1 "/bin/ryuk" About a minute ago Exited (0) 51 seconds ago testcontainers-ryuk-fb9e6553-bc15-41eb-8f23-adc66243251f
5bb48b21000e testcontainers/ryuk:0.5.1 "/bin/ryuk" About a minute ago Exited (0) 52 seconds ago testcontainers-ryuk-aff1a002-052b-48e7-9425-dd6e0d344abf
6ab93eb9bf98 testcontainers/ryuk:0.5.1 "/bin/ryuk" About a minute ago Exited (0) 53 seconds ago testcontainers-ryuk-1b6f4b61-859f-418b-8c09-f1e3069c795d
2846d9812f13 testcontainers/ryuk:0.5.1 "/bin/ryuk" About a minute ago Exited (0) 55 seconds ago testcontainers-ryuk-7fa34e33-0616-41a5-b357-b7ffef21b90d
f8c2b8105760 testcontainers/ryuk:0.5.1 "/bin/ryuk" About a minute ago Exited (0) 56 seconds ago testcontainers-ryuk-0dd023fd-06c8-40da-8072-f200c688cc80
d0b4c7c71c58 testcontainers/ryuk:0.5.1 "/bin/ryuk" About a minute ago Exited (0) 56 seconds ago testcontainers-ryuk-178ec9c2-85c2-47b1-a151-57ef5e6bff12
ccba9b4802c3 testcontainers/ryuk:0.5.1 "/bin/ryuk" About a minute ago Exited (0) About a minute ago testcontainers-ryuk-b1fb17dc-d87e-48d7-a34b-e3c42520b097
ef1231b2b12a testcontainers/ryuk:0.5.1 "/bin/ryuk" About a minute ago Exited (0) About a minute ago testcontainers-ryuk-97c367bc-e6c1-411d-aa97-57be26e2eb4e
921164cbb0c4 testcontainers/ryuk:0.5.1 "/bin/ryuk" 2 minutes ago Exited (0) About a minute ago testcontainers-ryuk-c7bdf927-b1a9-4ba2-b1ef-e5ae475a01ff
f44712b386a7 testcontainers/ryuk:0.5.1 "/bin/ryuk" 2 minutes ago Exited (0) About a minute ago testcontainers-ryuk-b613f053-ba7f-48a0-982d-bc737babd2b0
4391bf404514 testcontainers/ryuk:0.5.1 "/bin/ryuk" 2 minutes ago Exited (0) 2 minutes ago testcontainers-ryuk-2f00dec7-519e-45b0-915f-1dbc2bb05eab
1104b643b606 testcontainers/ryuk:0.5.1 "/bin/ryuk" 2 minutes ago Exited (0) 2 minutes ago testcontainers-ryuk-03761072-2a9e-439a-b0b5-a25a833ae55b
71aa1530031e testcontainers/ryuk:0.5.1 "/bin/ryuk" 3 minutes ago Exited (0) 2 minutes ago testcontainers-ryuk-9da38d7f-0e42-413f-81d5-742cd58d9820
f56a6a2e44f6 testcontainers/ryuk:0.5.1 "/bin/ryuk" 3 minutes ago Exited (0) 2 minutes ago testcontainers-ryuk-b244b3c3-4c5e-4fd7-a0c9-c7b0bc5cb51a
e8172012e041 testcontainers/ryuk:0.5.1 "/bin/ryuk" 3 minutes ago Exited (0) 3 minutes ago testcontainers-ryuk-d7d4b0b8-13d1-4c68-9bdb-8468cd85486c
79a37961a9cd testcontainers/ryuk:0.5.1 "/bin/ryuk" 3 minutes ago Exited (0) 3 minutes ago testcontainers-ryuk-67a2bd01-07e6-41ea-8d03-6b97e95597dd
f0e9749bf00c testcontainers/ryuk:0.5.1 "/bin/ryuk" 4 minutes ago Exited (0) 3 minutes ago testcontainers-ryuk-a590bd18-2a6e-4e85-8757-04bc76930e23
89336b1bc1d6 testcontainers/ryuk:0.5.1 "/bin/ryuk" 4 minutes ago Exited (0) 3 minutes ago testcontainers-ryuk-2c5bd5c6-643a-4607-870a-9eec0abbd7e5
ea324a16bc8c testcontainers/ryuk:0.5.1 "/bin/ryuk" 4 minutes ago Exited (0) 4 minutes ago testcontainers-ryuk-cfd431c4-bb50-4bed-8375-d0fc149b5780
fcd1566778f6 testcontainers/ryuk:0.5.1 "/bin/ryuk" 4 minutes ago Exited (0) 4 minutes ago testcontainers-ryuk-ee436a51-5ece-4122-b0f0-0401f9c4d1a5
fd77f7842f1f testcontainers/ryuk:0.5.1 "/bin/ryuk" 5 minutes ago Exited (0) 4 minutes ago testcontainers-ryuk-4967d460-d296-430b-b8c9-5fd343563eef
8e10e3de770f testcontainers/ryuk:0.5.1 "/bin/ryuk" 5 minutes ago Exited (0) 4 minutes ago testcontainers-ryuk-9261c4e2-e35e-4b9d-9184-13eb6d4f8e7e
1faf36416c86 testcontainers/ryuk:0.5.1 "/bin/ryuk" 5 minutes ago Exited (0) 5 minutes ago testcontainers-ryuk-e6a85573-f701-4de7-b503-f11b3f3b47cf
02bb029d442d testcontainers/ryuk:0.5.1 "/bin/ryuk" 5 minutes ago Exited (0) 5 minutes ago testcontainers-ryuk-542b0790-1673-4532-a281-8ecfae0c3174
f711663ce13e testcontainers/ryuk:0.5.1 "/bin/ryuk" 6 minutes ago Exited (0) 5 minutes ago testcontainers-ryuk-f903520b-d021-4ac5-98f7-6d88c486f232
7d136c8f8532 testcontainers/ryuk:0.5.1 "/bin/ryuk" 6 minutes ago Exited (0) 5 minutes ago testcontainers-ryuk-a59f1aac-6bd3-4e2a-bcde-0fdc0b2d839f
75114daca8ca testcontainers/ryuk:0.5.1 "/bin/ryuk" 6 minutes ago Exited (0) 6 minutes ago testcontainers-ryuk-89cd6f0f-245b-4499-b7d0-c59b66fed775
8a5f76e9b660 testcontainers/ryuk:0.5.1 "/bin/ryuk" 6 minutes ago Exited (0) 6 minutes ago testcontainers-ryuk-24a5875a-7c71-4c9d-8f92-b9cc4708ee25
a8ab0510105e testcontainers/ryuk:0.5.1 "/bin/ryuk" 7 minutes ago Exited (0) 6 minutes ago testcontainers-ryuk-14eb7d3c-1552-4db5-89ee-4669ff94e353
9757e6699598 testcontainers/ryuk:0.5.1 "/bin/ryuk" 7 minutes ago Exited (0) 6 minutes ago testcontainers-ryuk-e69fdc48-bda7-40cf-9ff7-ef0352648b5a
8ca4efc76b39 testcontainers/ryuk:0.5.1 "/bin/ryuk" 7 minutes ago Exited (0) 7 minutes ago testcontainers-ryuk-be6c05d8-c8eb-4d65-abaf-99d8882c4aa8
b001249b35d8 testcontainers/ryuk:0.5.1 "/bin/ryuk" 7 minutes ago Exited (0) 7 minutes ago testcontainers-ryuk-85cacd09-d414-4b17-9595-65413c17a0a8
f3038dab3c61 testcontainers/ryuk:0.5.1 "/bin/ryuk" 8 minutes ago Exited (0) 7 minutes ago testcontainers-ryuk-358bc1e5-b278-451c-b7f0-bfe393656634
7c6225374c36 testcontainers/ryuk:0.5.1 "/bin/ryuk" 8 minutes ago Exited (0) 7 minutes ago testcontainers-ryuk-ece119eb-09fd-4cb8-b992-ba14ef6a4fa3
dad0e439e20e testcontainers/ryuk:0.5.1 "/bin/ryuk" 8 minutes ago Exited (0) 8 minutes ago testcontainers-ryuk-d9233fe8-a132-4ae2-bf62-6e003812b727
517f1769e141 testcontainers/ryuk:0.5.1 "/bin/ryuk" 9 minutes ago Exited (0) 8 minutes ago testcontainers-ryuk-b329a687-9e63-4abe-aa98-5421a4bcc496
2dd542698248 testcontainers/ryuk:0.5.1 "/bin/ryuk" 9 minutes ago Exited (0) 8 minutes ago testcontainers-ryuk-4c861aaa-baf5-4599-8b1b-c67c99714f95
aa8e63e3042a testcontainers/ryuk:0.5.1 "/bin/ryuk" 9 minutes ago Exited (0) 9 minutes ago testcontainers-ryuk-c589f726-e0cf-4faa-b453-037cd642ca24
alexanderankin commented
@shoffmeister can you test with https://pypi.org/project/testcontainers/4.2.0rc1/ ? thanks! nevermind, this approach will not work yet.
alexanderankin commented
https://pypi.org/project/testcontainers/4.2.0rc2/ is up, tested a bit by me, maybe fixes the issue?
shoffmeister commented
@alexanderankin Many thanks for the very quick reaction!
4.2.0.rc2 has survived a few minutes of while true; do pytest; done without leaking the ryuk container.
There is one oddity, though: Ctrl+C does not abort the tests. To reproduce
pytest(just a simple one, no need for the infinite loop)- run a test suite which uses (my) Kafka container
Ctrl+C
//exp: test run aborts
//act: test run stops (with ryuk + Kafka container gone) and appears to hang dead
Right now I do not know how to minimally reproduce this, what the root cause of that is, and whether this is a regression - it may simply be due to timeouts that are too high (ryuk, my container).
I believe that the current state is strictly better than the previous state, so I'd suggest proceeding with the current state for now.
alexanderankin commented
@shoffmeister - after a day of experimenting i realized that the shutdown hooks are not necessary - and actually just how necessary the ryuk is in practice. im releasing 4.2.0 right now with only the necessary parts of the above fix, and not the unnecessary ones. thanks for reporting and helping to test!