AnsiballZ_setup.py left behind in container /tmp
jordemort opened this issue · 14 comments
When I build a container with ansible-bender it leaves behind an AnziballZ_setup.py in a directory in /tmp - it'd be nice if ansible-bender cleaned this up.
Wow, this is pretty interesting. Especially since /tmp is meant to be tmpfs.
Unfortunately, I can't reproduce, could you please show me your base image, playbook
and how you build the image? I tried to repro using ths
playbook
and alpine (docker.io/library/python:3-alpine) as a base.
I am using things in a rather weird way :D
Here's my playbook:
---
- name: Build a container from an imported stage1 debootstrap
hosts: all
vars:
ansible_bender:
cache_tasks: no
layering: no
tasks:
- name: Make debootstrap script safe for containers
replace:
path: /debootstrap/debian-common
regexp: '\bsetup_(proc|devices|dynamic_devices)\b'
replace: '/bin/true'
- name: Run second-stage debootstrap
command: /debootstrap/debootstrap --second-stage
# We aren't using the "apt" ansible module here because it wants to
# automatically install python3-apt
- name: Clean apt archives
command: /usr/bin/apt-get clean
- name: Clean apt lists
command: /usr/bin/apt-get update
I'm running it against a tarball created with debootstrap --foreign --variant=minbase and then imported with podman import.
My command-line is something like:
ansible-bender --verbose build --no-cache
--build-volumes /py:/py
--python-interpreter /py/bin/python3.7m
/app/ansible/debootstrap.yml
registry.local/stage1 registry.local/stage2
/py contains a Python 3.7 build linked to be independent of the OS.
I'm not seeing in the code where it's supposed to be creating the tmpfs for this. Supplying my own --build-volume for /tmp seems to break things:
22:01:58.782 [err] [dockto.local/stage2] 22:01:58.782 utils.py INFO running command: "['ansible-playbook', '-c', 'buildah', '-i', '/tmp/abqyhnh03g/inventory', '/app/ansible/.debootstrap-20190605-220158021541-bwtnlvgcfl.yaml']"
22:02:00.781 [dockto.local/stage2]
22:02:00.781 [dockto.local/stage2] PLAY [Build a container from an imported stage1 debootstrap] *******************
22:02:00.794 [dockto.local/stage2]
22:02:00.794 [dockto.local/stage2] TASK [Gathering Facts] *********************************************************
22:02:02.513 [dockto.local/stage2] An exception occurred during task execution. To see the full traceback, use -vvv. The error was: FileNotFoundError: [Errno 2] No such file or directory: b'/var/lib/containers/storage/overlay/449c01a69920441780b2135d91d445891e68c2efe6a442a4932d3cb5983dede3/merged/tmp/ansible-tmp-1559772120.8052845-280252822352247/AnsiballZ_setup.py'
A little bit more verbose debugging:
22:17:11.938 [err] [dockto.local/stage2] 22:17:11.938 utils.py DEBUG PLAY [Build a container from an imported stage1 debootstrap] *******************
22:17:11.947 [err] [dockto.local/stage2] 22:17:11.946 utils.py DEBUG
22:17:11.947 [err] [dockto.local/stage2] 22:17:11.946 utils.py DEBUG TASK [Gathering Facts] *********************************************************
22:17:11.947 [err] [dockto.local/stage2] 22:17:11.946 utils.py DEBUG task path: /app/ansible/.debootstrap-20190605-221709129728-lwunvbocgs.yaml:1
22:17:11.958 [err] [dockto.local/stage2] 22:17:11.957 utils.py DEBUG <dockto-local-stage2-20190605-221706442286-cont> RUN [b'buildah', b'mount', b'--', b'dockto-local-stage2-20190605-221706442286-cont']
22:17:12.162 [err] [dockto.local/stage2] 22:17:12.162 utils.py DEBUG <dockto-local-stage2-20190605-221706442286-cont> RUN [b'buildah', b'run', b'--', b'dockto-local-stage2-20190605-221706442286-cont', b'/bin/sh', b'-c', b'( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1559773031.9569387-26640349908499 `" && echo ansible-tmp-1559773031.9569387-26640349908499="` echo /tmp/ansible-tmp-1559773031.9569387-26640349908499 `" ) && sleep 0']
22:17:13.445 [err] [dockto.local/stage2] 22:17:13.445 utils.py DEBUG Using module file /opt/dockto/lib/python3.7/site-packages/ansible/modules/system/setup.py
22:17:13.448 [err] [dockto.local/stage2] 22:17:13.447 utils.py DEBUG <dockto-local-stage2-20190605-221706442286-cont> PUT /root/.ansible/tmp/ansible-local-498u9b2l632/tmpdqpurity TO /tmp/ansible-tmp-1559773031.9569387-26640349908499/AnsiballZ_setup.py
22:17:13.459 [err] [dockto.local/stage2] 22:17:13.458 utils.py DEBUG <dockto-local-stage2-20190605-221706442286-cont> RUN [b'buildah', b'umount', b'--', b'dockto-local-stage2-20190605-221706442286-cont']
22:17:13.650 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG The full traceback is:
22:17:13.650 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG Traceback (most recent call last):
22:17:13.650 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG File "/opt/dockto/lib/python3.7/site-packages/ansible/executor/task_executor.py", line 144, in run
22:17:13.650 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG res = self._execute()
22:17:13.650 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG File "/opt/dockto/lib/python3.7/site-packages/ansible/executor/task_executor.py", line 648, in _execute
22:17:13.650 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG result = self._handler.run(task_vars=variables)
22:17:13.651 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG File "/opt/dockto/lib/python3.7/site-packages/ansible/plugins/action/gather_facts.py", line 66, in run
22:17:13.651 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG res = self._execute_module(module_name=fact_module, module_args=mod_args, task_vars=task_vars, wrap_async=False)
22:17:13.651 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG File "/opt/dockto/lib/python3.7/site-packages/ansible/plugins/action/__init__.py", line 836, in _execute_module
22:17:13.651 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG self._transfer_data(remote_module_path, module_data)
22:17:13.651 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG File "/opt/dockto/lib/python3.7/site-packages/ansible/plugins/action/__init__.py", line 447, in _transfer_data
22:17:13.651 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG self._transfer_file(afile, remote_path)
22:17:13.651 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG File "/opt/dockto/lib/python3.7/site-packages/ansible/plugins/action/__init__.py", line 424, in _transfer_file
22:17:13.651 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG self._connection.put_file(local_path, remote_path)
22:17:13.652 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG File "/opt/dockto/lib/python3.7/site-packages/ansible/plugins/connection/buildah.py", line 136, in put_file
22:17:13.652 [err] [dockto.local/stage2] 22:17:13.649 utils.py DEBUG to_bytes(real_out_path, errors='surrogate_or_strict')
22:17:13.652 [err] [dockto.local/stage2] 22:17:13.650 utils.py DEBUG File "/opt/dockto/lib/python3.7/shutil.py", line 121, in copyfile
22:17:13.652 [err] [dockto.local/stage2] 22:17:13.650 utils.py DEBUG with open(dst, 'wb') as fdst:
22:17:13.652 [err] [dockto.local/stage2] 22:17:13.650 utils.py DEBUG FileNotFoundError: [Errno 2] No such file or directory: b'/var/lib/containers/storage/overlay/dbce39253c16b325e14b340821536f3950f110d985d4703e30a4c0b5970d11b5/merged/tmp/ansible-tmp-1559773031.9569387-26640349908499/AnsiballZ_setup.py'
22:17:13.652 [err] [dockto.local/stage2] 22:17:13.650 utils.py DEBUG fatal: [dockto-local-stage2-20190605-221706442286-cont]: FAILED! => {
22:17:13.653 [err] [dockto.local/stage2] 22:17:13.650 utils.py DEBUG "msg": "Unexpected failure during module execution.",
22:17:13.653 [err] [dockto.local/stage2] 22:17:13.650 utils.py DEBUG "stdout": ""
22:17:13.653 [err] [dockto.local/stage2] 22:17:13.650 utils.py DEBUG }
Looks like buildah mount only mounts the container root, and not anything else that might be mounted in the container, so my build volume on /tmp masks what you're trying to copy in.
This is a buildah bug, IMO: containers/buildah#1509
I wonder if put_file in the ansible buildah connection driver could be reimplemented in terms of buildah copy...
yeah, utilizing buildah copy would be the best solution; I wonder what would be the easiest solution here - should bender try to remove remove the AnsiballZ_*.py files before committing a layer?
Not sure, I think there might be something broken somewhere else:
23:09:19.080 [err] [dockto.local/base] 23:09:19.080 utils.py DEBUG 23:09:19.079 db.py DEBUG runtime dir is /var/tmp
23:09:19.092 [err] [dockto.local/base] 23:09:19.094 utils.py DEBUG <dockto-local-base-20190606-230009674553-cont> RUN [b'buildah', b'mount', b'--', b'dockto-local-base-20190606-230009674553-cont']
23:09:19.375 [err] [dockto.local/base] 23:09:19.378 utils.py DEBUG <dockto-local-base-20190606-230009674553-cont> RUN [b'buildah', b'run', b'--', b'dockto-local-base-20190606-230009674553-cont', b'/bin/sh', b'-c', b'( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1559862559.0941648-8558565515363 `" && echo ansible-tmp-1559862559.0941648-8558565515363="` echo /tmp/ansible-tmp-1559862559.0941648-8558565515363 `" ) && sleep 0']
23:09:20.520 [err] [dockto.local/base] 23:09:20.522 utils.py DEBUG Using module file /opt/dockto/lib/python3.7/site-packages/ansible/modules/commands/command.py
23:09:20.520 [err] [dockto.local/base] 23:09:20.523 utils.py DEBUG <dockto-local-base-20190606-230009674553-cont> PUT /root/.ansible/tmp/ansible-local-64987t8adps/tmplsvt28sy TO /tmp/ansible-tmp-1559862559.0941648-8558565515363/AnsiballZ_command.py
23:09:20.523 [err] [dockto.local/base] 23:09:20.525 utils.py DEBUG <dockto-local-base-20190606-230009674553-cont> RUN [b'buildah', b'run', b'--', b'dockto-local-base-20190606-230009674553-cont', b'/bin/sh', b'-c', b'chmod u+x /tmp/ansible-tmp-1559862559.0941648-8558565515363/ /tmp/ansible-tmp-1559862559.0941648-8558565515363/AnsiballZ_command.py && sleep 0']
23:09:21.654 [err] [dockto.local/base] 23:09:21.656 utils.py DEBUG <dockto-local-base-20190606-230009674553-cont> RUN [b'buildah', b'run', b'--', b'dockto-local-base-20190606-230009674553-cont', b'/bin/sh', b'-c', b'/opt/dockto/bin/python3.7m /tmp/ansible-tmp-1559862559.0941648-8558565515363/AnsiballZ_command.py && sleep 0']
23:09:22.805 [err] [dockto.local/base] 23:09:22.807 utils.py DEBUG <dockto-local-base-20190606-230009674553-cont> RUN [b'buildah', b'run', b'--', b'dockto-local-base-20190606-230009674553-cont', b'/bin/sh', b'-c', b'rm -f -r /tmp/ansible-tmp-1559862559.0941648-8558565515363/ > /dev/null 2>&1 && sleep 0']
23:09:23.861 [err] [dockto.local/base] 23:09:23.863 utils.py DEBUG <dockto-local-base-20190606-230009674553-cont> RUN [b'buildah', b'umount', b'--', b'dockto-local-base-20190606-230009674553-cont']
I definitely see Ansible trying to clean up the tempdir after every play; the file really shouldn't survive that. I'm not sure why that isn't turning out to be effective.
Is this ansible 2.8 or 2.7? Maybe something changed in 2.8 which results into this odd behaviour.
This is 2.8
If I could get pipelining to work I think that would make this moot but it doesn't work right now, I think because of containers/buildah#1650
Looking at this a bit more closely, it appears that only the AnsiballZ_setup.py from the initial "Gathering Facts" play is left behind. It doesn't appear that Ansible is even trying to clean that one up. The other ones appear to be getting cleaned up properly. What happens if I turn off fact gathering for ansible-bender?
Turning off fact gathering leaves me with a clean /tmp and seems to have had no adverse effect on my use case.
Sounds like a bug in "ansible 2.8". Glad you were able to resolve it.
Btw Bender does not require any variables from the fact gathering, so it's really up to you if you want to run setup.