Docker orphan containers not being removed run out of storage
noahpro99 opened this issue · 3 comments
failed: [3.144.201.10] (item=apt-transport-https) => {"ansible_loop_var": "item", "changed": false, "item": "apt-transport-https", "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 107, in <module>\n File \"<stdin>\", line 99, in _ansiballz_main\n File \"<stdin>\", line 47, in invoke_module\n File \"/usr/lib/python3.10/runpy.py\", line 224, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib/python3.10/runpy.py\", line 96, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_apt_payload_7tku6j_z/ansible_apt_payload.zip/ansible/modules/apt.py\", line 1487, in <module>\n File \"/tmp/ansible_apt_payload_7tku6j_z/ansible_apt_payload.zip/ansible/modules/apt.py\", line 1358, in main\n File \"/usr/lib/python3/dist-packages/apt/cache.py\", line 214, in open\n self._cache = apt_pkg.Cache(progress)\napt_pkg.Error: E:Write error - write (28: No space left on device), E:IO Error saving source cache, E:The package lists or status file could not be parsed or opened.\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
failed: [3.144.201.10] (item=ca-certificates) => {"ansible_loop_var": "item", "changed": false, "item": "ca-certificates", "msg": "E:Write error - write (28: No space left on device), E:IO Error saving source cache, E:The package lists or status file could not be parsed or opened."}
failed: [3.144.201.10] (item=curl) => {"ansible_loop_var": "item", "changed": false, "item": "curl", "msg": "E:Write error - write (28: No space left on device), E:IO Error saving source cache, E:The package lists or status file could not be parsed or opened."}
failed: [3.144.201.10] (item=software-properties-common) => {"ansible_loop_var": "item", "changed": false, "item": "software-properties-common", "msg": "E:Write error - write (28: No space left on device), E:IO Error saving source cache, E:The package lists or status file could not be parsed or opened."}
failed: [3.144.201.10] (item=python3-pip) => {"ansible_loop_var": "item", "changed": false, "item": "python3-pip", "msg": "E:Write error - write (28: No space left on device), E:IO Error saving source cache, E:The package lists or status file could not be parsed or opened."}
failed: [3.144.201.10] (item=virtualenv) => {"ansible_loop_var": "item", "changed": false, "item": "virtualenv", "msg": "E:Write error - write (28: No space left on device), E:IO Error saving source cache, E:The package lists or status file could not be parsed or opened."}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 28] No space left on device
failed: [3.144.201.10] (item=python3-setuptools) => {"ansible_loop_var": "item", "changed": false, "item": "python3-setuptools", "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 107, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\nOSError: [Errno 28] No space left on device\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
PLAY RECAP *********************************************************************
3.144.201.10 : ok=2 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
2023-07-21 04:35:30,450 root WARNING
~#~#~#~DEPLOYING OPS REPO [ansible] FAILED~#~#~#~
Error: Process completed with exit code 2.
I deduced this is from not running docker compose with --remove-orphans. Is there some way to make that happen?
ubuntu@server:~/repo$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 17 8 4.027GB 3.089GB (76%)
Containers 8 6 320.4MB 319.7MB (99%)
Local Volumes 9 4 316MB 185.6kB (0%)
Build Cache 18 0 647.5kB 647.5kB
ubuntu@server:~/repo$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> 981c4d98ab9f About an hour ago 128MB
backend latest 69f74e05ef24 About an hour ago 447MB
<none> <none> 04ff41bedb32 About an hour ago 447MB
celeryworker latest 2ec687df1aed 2 hours ago 447MB
celerybeat latest 78073bb3927b 2 hours ago 447MB
<none> <none> 39a6495bb48a 2 hours ago 447MB
<none> <none> a38d1d3a7e0b 2 hours ago 447MB
<none> <none> 3ae00a0f998f 2 hours ago 447MB
<none> <none> 6a8d8f2caa93 2 hours ago 447MB
<none> <none> 8740d113c46c 3 hours ago 447MB
<none> <none> 3815a8f85297 3 hours ago 447MB
<none> <none> 0add46c8c7ba 3 hours ago 447MB
rabbitmq 3 bcef1e37beb2 3 days ago 226MB
mongo latest fb5fba25b25a 7 days ago 654MB
traefik v2.10 a07d037aae0a 4 weeks ago 144MB
mher/flower latest b749493444bc 4 weeks ago 85.4MB
python 3.11.4-slim-bullseye e7e77196c2b2 5 weeks ago 128MB
Hi @noahpro99 , first of all, thanks for opening an issue with us!
There are a few options at the moment that might help you overcome this issue, but none of them is exactly just a --remove-orphans option.
So, options.
- docker_full_cleanup - That option was introduced with the idea to be able to start from scratch, removing anything docker, ensuring a clean without redoing the whole vm.
- ec2_volume_size - You can always expand your disk a little bit, some deployments won't fit in 8 Gb.
Now, adding the remove_orphans option is something we can add to our list. Not sure how many users would use it, but it's always good to have another option.
Let me know how that goes, and will update you when we make that remove_orphan_option available.
Leo
Hi Leo,
Thanks for replying! I should have pointed out that all of the images on the server are orphans from running the action multiple times in a row. They just build up every time the action runs. That is how I eventually get the [Errno 28] No space left on device. The full cleanup would work if I wasn't using volumes but I am, unfortunately. For now, I log in to the server every few runs to run the --remove-orphans manually. I will stay tuned if you do get around to it.
Thanks,
Noah
Hi @noahpro99
But it is solved already! See the latest release.
The option is part of this group.. Test it out, and let us know if anything comes up.
Leo