ansible/molecule

More "Failed to create temporary directory."

J-SirL opened this issue · 5 comments

J-SirL commented

I have run into the same issue but using RHEL ubi and podman driver.
I am using rotless podman with molecule version 4.0.1

$ molecule --version
molecule 4.0.1 using python 3.8
ansible:2.13.1
delegated:4.0.1 from molecule
podman:2.0.2 from molecule_podman requiring collections: containers.podman>=1.7.0 ansible.posix>=1.3.0

--
Any tips on how to solve the issue would be appreciated

Discussed in #3092

Originally posted by lgadalla-impinj April 8, 2021
I've been struggling with the converge issue where Ansible cannot gather facts, allegedly due to not having permissions to write to the temporary directory:

TASK [Gathering Facts] *********************************************************
fatal: [instance]: UNREACHABLE! => changed=false
  msg: 'Failed to create temporary directory.In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in "/tmp", for more error information use -vvv. Failed command was: ( umask 77 && mkdir -p "` echo /tmp `"&& mkdir "` echo /tmp/ansible-tmp-1617853604.764122-263554-181361534476501 `" && echo ansible-tmp-1617853604.764122-263554-181361534476501="` echo /tmp/ansible-tmp-1617853604.764122-263554-181361534476501 `" ), exited with result 1'
  unreachable: true

The environment is as follows:

$ molecule --version
molecule 3.2.2 using python 3.8
    ansible:2.10.5
    delegated:3.2.2 from molecule
    docker:0.3.3 from molecule_docker

What is interesting is that when I run with two closely related OS Docker images, CentOS 7 and RHEL 7 in this case, the problem does not occur with CentOS 7 but does occur with RHEL 7. I've left remote_tmp set to /tmp but that does not seem to have any effect.

I'm mostly looking for any clues or guidance on how to dig further down and debug this issue.

apatard commented

iirc facts gathering is disabled on create, so you may want to:

  • run molecule create to create test containers
  • run molecule login to connect to the test containers
  • check /tmp rights to see if something wrong
  • if nothing detected, run molecule converge or molecule converge -vvvv to get the error and then log into the failing container with molecule login. Look at the logs in order to check if something interesting is in them.

I can't say more without having experienced the issue locally. Look for similar issues with ansible since it may not be a molecule specific error.

At worse, you may try to workaround things by mounting /tmp with a tmpfs:

platforms:
- name: instance
  image: myimage
  tmpfs:
  - /tmp
mr-scripting commented

@apatard I'm having a similar issue using wsl2 on windows.
molecule create is able to create the instance but when I execute molecule converge it fails with the error:
fatal: [instance]: FAILED! => { "msg": "failed to resolve remote temporary directory from ansible-tmp-1670169152.6252987-3617-243105984205172: ( umask 77 && mkdir -p "echo ~/.ansible/tmp"&& mkdir "echo ~/.ansible/tmp/ansible-tmp-1670169152.6252987-3617-243105984205172" && echo ansible-tmp-1670169152.6252987-3617-243105984205172="echo ~/.ansible/tmp/ansible-tmp-1670169152.6252987-3617-243105984205172" ) returned empty string" }

I'm able to login into the container instance with molecule login and I'm able to see the directories that are created:
root@instance:~/.ansible/tmp# ls ansible-tmp-1670168194.9145443-21993-117827900798 ansible-tmp-1670169152.6252987-3617-243105984205172 ansible-tmp-1670168967.0483384-3291-64050985022207 ansible-tmp-1670169174.0279183-3801-267177116955287

The directories are empty though.
How to proceed from here?

molecule 4.0.3 using python 3.10
ansible:2.13.4
delegated:4.0.3 from molecule
docker:2.1.0 from molecule_docker requiring collections: community.docker>=3.0.2 ansible.posix>=1.4.0

cat /etc/issue
Ubuntu 22.04.1 LTS \n \l

wsl --version
WSL version: 1.0.0.0
Kernel version: 5.15.74.2
WSLg version: 1.0.47
MSRDC version: 1.2.3575
Direct3D version: 1.606.4
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22621.819

  • 👍2
javanaut-de commented

This appears to happen using podman and docker drivers with a similar failure mode: #3818

calillo commented

I have similar issue on Mac (but also on Windows it's the same) when running molecule inside container (with Docker Desktop). After spending a lot of time trying downgrading versions I realized that the cause was to be found elsewhere and found it in Docker Desktop, in fact natively on Mac (or on the Windows WSL) it worked perfectly.
At first, the reasoning was not clear to me but after reading some kernel intuition I realized it could be something like this. In fact months ago everything works and at some point it stopped and returning the error "failed to resolve remote temporary directory ... returned empty string"
From here I found out that since Docker Desktop 4.13 they upgraded the kernel from 5.10.x to 5.15.x and this is the root cause. Downgrading Docker Desktop to 4.12 on both Mac and Windows everything started working again. I think this problem also exists on a Linux distribution with a kernel 5.15.x, I can't tell if it's an incompatibility between ansible/molecule and the kernel. I hope someone fix this issue.

calillo commented

I have similar issue on Mac (but also on Windows it's the same) when running molecule inside container (with Docker Desktop). After spending a lot of time trying downgrading versions I realized that the cause was to be found elsewhere and found it in Docker Desktop, in fact natively on Mac (or on the Windows WSL) it worked perfectly.
At first, the reasoning was not clear to me but after reading some kernel intuition I realized it could be something like this. In fact months ago everything works and at some point it stopped and returning the error "failed to resolve remote temporary directory ... returned empty string"
From here I found out that since Docker Desktop 4.13 they upgraded the kernel from 5.10.x to 5.15.x and this is the root cause. Downgrading Docker Desktop to 4.12 on both Mac and Windows everything started working again. I think this problem also exists on a Linux distribution with a kernel 5.15.x, I can't tell if it's an incompatibility between ansible/molecule and the kernel. I hope someone fix this issue.