Ansible Interpreter discovery not working with pyenv/poetry/molecule/docker
timblaktu opened this issue · 4 comments
timblaktu commented
Issue Type
- Bug report
Molecule and Ansible details
> ansible --version && molecule --version
ansible [core 2.12.6]
config file = /home/tim/src/timblaktu-cm/ansible/ansible.cfg
configured module search path = ['/home/tim/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /home/tim/.cache/pypoetry/virtualenvs/timblaktu-cm-ansible-X34RCiD6-py3.10/lib/python3.10/site-packages/ansible
ansible collection location = /home/tim/.ansible/collections:/usr/share/ansible/collections
executable location = /home/tim/.cache/pypoetry/virtualenvs/timblaktu-cm-ansible-X34RCiD6-py3.10/bin/ansible
python version = 3.10.0 (default, Oct 12 2021, 11:25:19) [GCC 8.3.0]
jinja version = 3.1.2
libyaml = True
molecule 4.0.0 using python 3.10
ansible:2.12.6
delegated:4.0.0 from molecule
docker:1.1.0 from molecule_docker requiring collections: community.docker>=1.9.1
Molecule installation method (one of):
- poetry
Ansible installation method (one of):
- poetry
Summary
As noted over in ansible-community/molecule-docker#151 and ansible/ansible#78090, ansible is not able to find the python interpreter when run from a simple "deb10-headless" molecule scenario using docker driver:
---
dependency:
name: galaxy
driver:
name: docker
platforms:
- name: instance
image: debian:buster
pre_build_image: true
provisioner:
name: ansible
verifier:
name: ansible # TODO: pytest-testinfraAfter successfully creating the container using molecule create --scenario-name deb10-headless, and confirming success with:
> molecule list --scenario-name deb10-headless
INFO Running deb10-headless > list
╷ ╷ ╷ ╷ ╷
Instance Name │ Driver Name │ Provisioner Name │ Scenario Name │ Created │ Converged
╶───────────────┼─────────────┼──────────────────┼────────────────┼─────────┼───────────╴
instance │ docker │ ansible │ deb10-headless │ true │ false
╵ ╵ ╵ ╵ ╵
> docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
252bf11e0d1d debian:buster "bash -c 'while true…" 2 hours ago Up 2 hours instance
I get error output from molecule --verbose converge --scenario-name deb10-headless:
TASK [Gathering Facts] *********************************************************
task path: /home/tim/src/timblaktu-cm/ansible/tblack-deb10-headless.yml:8
<instance> ESTABLISH DOCKER CONNECTION FOR USER: root
<instance> EXEC ['/usr/bin/docker', '-H=unix:///mnt/wsl/shared-docker/docker.sock', b'exec', b'-i', 'instance', '/bin/sh', '-c', "/bin/sh -c 'echo ~ && sleep 0'"]
<instance> EXEC ['/usr/bin/docker', '-H=unix:///mnt/wsl/shared-docker/docker.sock', b'exec', b'-i', 'instance', '/bin/sh', '-c', '/bin/sh -c \'( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1655759232.9442172-20889-87937448235310 `" && echo ansible-tmp-1655759232.9442172-20889-87937448235310="` echo /root/.ansible/tmp/ansible-tmp-1655759232.9442172-20889-87937448235310 `" ) && sleep 0\'']
Using module file /home/tim/.cache/pypoetry/virtualenvs/timblaktu-cm-ansible-X34RCiD6-py3.10/lib/python3.10/site-packages/ansible/modules/setup.py
<instance> PUT /home/tim/.ansible/tmp/ansible-local-2088466y_h2kc/tmppuqtmyv1 TO /root/.ansible/tmp/ansible-tmp-1655759232.9442172-20889-87937448235310/AnsiballZ_setup.py
<instance> EXEC ['/usr/bin/docker', '-H=unix:///mnt/wsl/shared-docker/docker.sock', b'exec', b'-i', 'instance', '/bin/sh', '-c', "/bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1655759232.9442172-20889-87937448235310/ /root/.ansible/tmp/ansible-tmp-1655759232.9442172-20889-87937448235310/AnsiballZ_setup.py && sleep 0'"]
<instance> EXEC ['/usr/bin/docker', '-H=unix:///mnt/wsl/shared-docker/docker.sock', b'exec', b'-i', 'instance', '/bin/sh', '-c', "/bin/sh -c '/home/tim/.cache/pypoetry/virtualenvs/timblaktu-cm-ansible-X34RCiD6-py3.10/bin/python /root/.ansible/tmp/ansible-tmp-1655759232.9442172-20889-87937448235310/AnsiballZ_setup.py && sleep 0'"]
<instance> EXEC ['/usr/bin/docker', '-H=unix:///mnt/wsl/shared-docker/docker.sock', b'exec', b'-i', 'instance', '/bin/sh', '-c', "/bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1655759232.9442172-20889-87937448235310/ > /dev/null 2>&1 && sleep 0'"]
fatal: [instance]: FAILED! => {
"ansible_facts": {},
"changed": false,
"failed_modules": {
"ansible.legacy.setup": {
"failed": true,
"module_stderr": "/bin/sh: 1: /home/tim/.cache/pypoetry/virtualenvs/timblaktu-cm-ansible-X34RCiD6-py3.10/bin/python: not found\n",
"module_stdout": "",
"msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error",
"rc": 127
}
},
"msg": "The following modules failed to execute: ansible.legacy.setup\n"
}
PLAY RECAP *********************************************************************
instance : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
WARNING Retrying execution failure 2 of: ansible-playbook --inventory /home/tim/.cache/molecule/ansible/deb10-headless/inventory --skip-tags molecule-notest,notest /home/tim/src/timblaktu-cm/ansible/molecule/deb10-headless/converge.yml
CRITICAL Ansible return code was 2, command was: ['ansible-playbook', '--inventory', '/home/tim/.cache/molecule/ansible/deb10-headless/inventory', '--skip-tags', 'molecule-notest,notest', '/home/tim/src/timblaktu-cm/ansible/molecule/deb10-headless/converge.yml']
Even though I'm telling ansible explicitly this path using:
export ANSIBLE_PYTHON_INTERPRETER="$(which python)"
(and if I don't specify this env var, the error still happens but it complains about the path /usr/bin/python.)
The active python interpreter on my path is a poetry venv shim:
> which python
/home/tim/.cache/pypoetry/virtualenvs/timblaktu-cm-ansible-X34RCiD6-py3.10/bin/python
.. which points to a pyenv-installed interpreter binary:
> ll /home/tim/.cache/pypoetry/virtualenvs/timblaktu-cm-ansible-X34RCiD6-py3.10/bin/python
lrwxrwxrwx 1 tim tim 47 Jun 18 11:21 /home/tim/.cache/pypoetry/virtualenvs/timblaktu-cm-ansible-X34RCiD6-py3.10/bin/python -> /home/tim/.pyenv/versions/3.10.0/bin/python3.10*
I suspect this is a molecule issue (or perhaps me misusing molecule) because I can successfully run ansible in this environment to gather localhost facts:
> ansible -m setup localhost (1 results) [1047/64866][WARNING]: No inventory was parsed, only implicit localhost is available
localhost | SUCCESS => {
"ansible_facts": {
.
.
I understand from ansible/ansible#74045 that this is thought to be caused by an implicit localhost, but since I'm specifying ANSIBLE_PYTHON_INTERPRETER explicitly, and I'm using the molecule-docker driver which generates an inventory file to pass to the playbook, and since the error I'm getting clearly indicates ansible is using this path, I believe this to be another problem.
Ansible Version
$ ansible --version
ansible [core 2.12.6]
config file = /home/tim/src/timblaktu-cm/ansible/ansible.cfg
configured module search path = ['/home/tim/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /home/tim/.cache/pypoetry/virtualenvs/timblaktu-cm-ansible-X34RCiD6-py3.10/lib/python3.10/site-packages/ansible
ansible collection location = /home/tim/.ansible/collections:/usr/share/ansible/collections
executable location = /home/tim/.cache/pypoetry/virtualenvs/timblaktu-cm-ansible-X34RCiD6-py3.10/bin/ansible
python version = 3.10.0 (default, Oct 12 2021, 11:25:19) [GCC 8.3.0]
jinja version = 3.1.2
libyaml = TrueAnsible Configuration
# if using a version older than ansible-core 2.12 you should omit the '-t all'
$ ansible-config dump --only-changed -t all
DEFAULT_FORKS(/home/tim/src/timblaktu-cm/ansible/ansible.cfg) = 20
DEFAULT_LOG_PATH(/home/tim/src/timblaktu-cm/ansible/ansible.cfg) = /home/tim/src/timblaktu-cm/ansible/ansible.log
DEFAULT_MANAGED_STR(/home/tim/src/timblaktu-cm/ansible/ansible.cfg) = #Ansible managed by user {uid} on {host} from template file:
DEFAULT_POLL_INTERVAL(/home/tim/src/timblaktu-cm/ansible/ansible.cfg) = 5
DEFAULT_ROLES_PATH(/home/tim/src/timblaktu-cm/ansible/ansible.cfg) = ['/etc/ansible/roles', '/home/tim/src/timblaktu-cm/ansible/roles', '/home/tim/.ansible/collections/ansib>DEFAULT_STDOUT_CALLBACK(/home/tim/src/timblaktu-cm/ansible/ansible.cfg) = debug
HOST_KEY_CHECKING(/home/tim/src/timblaktu-cm/ansible/ansible.cfg) = False
INTERPRETER_PYTHON(env: ANSIBLE_PYTHON_INTERPRETER) = /home/tim/.cache/pypoetry/virtualenvs/timblaktu-cm-ansible-X34RCiD6-py3.10/bin/python
RETRY_FILES_ENABLED(/home/tim/src/timblaktu-cm/ansible/ansible.cfg) = False
BECOME:
======
CACHE:
=====
CALLBACK:
========
CLICONF:
=======
CONNECTION:
==========
paramiko_ssh:
____________
host_key_checking(/home/tim/src/timblaktu-cm/ansible/ansible.cfg) = False
ssh:
___
host_key_checking(/home/tim/src/timblaktu-cm/ansible/ansible.cfg) = False
pipelining(/home/tim/src/timblaktu-cm/ansible/ansible.cfg) = False
ssh_args(/home/tim/src/timblaktu-cm/ansible/ansible.cfg) = -o ControlMaster=auto -o ControlPersist=60s -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no
HTTPAPI:
=======
INVENTORY:
=========
LOOKUP:
======
NETCONF:
=======
SHELL:
=====
sh:
__
world_readable_temp(/home/tim/src/timblaktu-cm/ansible/ansible.cfg) = True
VARS:
====OS / Environment
Debian 11.2 in WSL
Virtual Environment Versions:
> poetry show
ansible 5.9.0 Radically simple IT automation
ansible-compat 2.1.0 Ansible compatibility goodies
ansible-core 2.12.6 Radically simple IT automation
arrow 1.2.2 Better dates & times for Python
attrs 21.4.0 Classes Without Boilerplate
binaryornot 0.4.4 Ultra-lightweight pure Python package to check if a file is binary or text.
cerberus 1.3.2 Lightweight, extensible schema and data validation tool for Python dictionaries.
certifi 2022.6.15 Python package for providing Mozilla's CA Bundle.
cffi 1.15.0 Foreign Function Interface for Python calling C code.
chardet 4.0.0 Universal encoding detector for Python 2 and 3
charset-normalizer 2.0.12 The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet.
click 8.1.3 Composable command line interface toolkit
click-help-colors 0.9.1 Colorization of help messages in Click
commonmark 0.9.1 Python parser for the CommonMark Markdown spec
cookiecutter 2.1.1 A command-line utility that creates projects from project templates, e.g. creating a Python package project from a Python package project te...
cryptography 37.0.2 cryptography is a package which provides cryptographic recipes and primitives to Python developers.
distro 1.7.0 Distro - an OS platform information API
docker 5.0.3 A Python library for the Docker Engine API.
enrich 1.2.7 enrich
idna 3.3 Internationalized Domain Names in Applications (IDNA)
iniconfig 1.1.1 iniconfig: brain-dead simple config-ini parsing
jinja2 3.1.2 A very fast and expressive template engine.
jinja2-time 0.2.0 Jinja2 Extension for Dates and Times
jsonschema 4.6.0 An implementation of JSON Schema validation for Python
markupsafe 2.1.1 Safely add untrusted strings to HTML/XML markup.
molecule 4.0.0 Molecule aids in the development and testing of Ansible roles
molecule-docker 1.1.0 Molecule aids in the development and testing of Ansible roles
packaging 21.3 Core utilities for Python packages
pluggy 1.0.0 plugin and hook calling mechanisms for python
py 1.11.0 library with cross-python path, ini-parsing, io, code, log facilities
pycparser 2.21 C parser in Python
pygments 2.12.0 Pygments is a syntax highlighting package written in Python.
pyparsing 3.0.9 pyparsing module - Classes and methods to define and execute parsing grammars
pyrsistent 0.18.1 Persistent/Functional/Immutable data structures
pytest 7.1.2 pytest: simple powerful testing with Python
python-dateutil 2.8.2 Extensions to the standard Python datetime module
python-slugify 6.1.2 A Python slugify application that also handles Unicode
pyyaml 6.0 YAML parser and emitter for Python
requests 2.28.0 Python HTTP for Humans.
resolvelib 0.5.5 Resolve abstract dependencies into concrete ones
rich 12.4.4 Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal
selinux 0.2.1 shim selinux module
six 1.16.0 Python 2 and 3 compatibility utilities
subprocess-tee 0.3.5 subprocess-tee
text-unidecode 1.3 The most basic Text::Unidecode port
tomli 2.0.1 A lil' TOML parser
urllib3 1.26.9 HTTP library with thread-safe connection pooling, file post, and more.
websocket-client 1.3.2 WebSocket client for Python with low level API options
Steps to Reproduce
Inside my poetry shell, run molecule converge on the molecule scenario containing this test playbook:
---
- name: Converge
hosts: all
tasks:
- name: "Import Main Playbook to be Tested"
import_playbook:
name: "../../some-other-playbook-to-test.yml"Expected Results
I expected ansible to be able to find and run the python interpreter pointer to by ANSIBLE_INTERPRETER_PATH, which i set explicitly using:
export ANSIBLE_PYTHON_INTERPRETER="$(which python)"
which works just fine manually in the same shell I ran molecule from:
> /home/tim/.cache/pypoetry/virtualenvs/timblaktu-cm-ansible-X34RCiD6-py3.10/bin/python
Python 3.10.0 (default, Oct 12 2021, 11:25:19) [GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
Actual Results
TASK [Gathering Facts] *********************************************************
fatal: [instance]: FAILED! => {"ansible_facts": {}, "changed": false, "failed_modules": {"ansible.legacy.setup": {"failed": true, "module_stderr": "/bin/sh: 1: /home/tim/.cache/pypoetry/virtualenvs/timblaktu-cm-ansible-X34RCiD6-py3.10/bin/python: not found\n", "module_stdout": "", "msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error", "rc": 127}}, "msg": "The following modules failed to execute: ansible.legacy.setup\n"}zhan9san commented
If you want to import a playbook, files with a list of plays can only be included at the top level.
Please note the indent.
You cannot use this action inside a play.
Here is an example.
- hosts: localhost
tasks:
- debug:
msg: play1
- name: Include a play after another play
import_playbook: otherplays.yaml
- name: Set variables on an imported playbook
import_playbook: otherplays.yml
vars:
service: httpd
- name: This DOES NOT WORK
hosts: all
tasks:
- debug:
msg: task1
- name: This fails because I'm inside a play already
import_playbook: stuff.yaml
How to test
~/src/tmp/my_new_role_docker 11s 23:17:52
v-397-test ❯ cat molecule/default/converge.yml
---
- name: Converge
hosts: all
tasks:
- name: "Include acme.my_new_role_docker"
include_role:
name: "acme.my_new_role_docker"
- name: Import Main Playbook to be Tested
import_playbook: ../../a.yml
~/src/tmp/my_new_role_docker 23:23:16
v-397-test ❯ cat a.yml
---
- name: Converge
hosts: all
tasks:
- name: "Include acme.my_new_role_docker"
debug:
msg: "{{ molecule_ephemeral_directory | default(5) }}"
~/src/tmp/my_new_role_docker 23:23:23
v-397-test ❯ cat a.yml
~/src/tmp/my_new_role_docker 23:23:26
v-397-test ❯ molecule converge
[WARNING]: You are running the development version of Ansible. You should only
run Ansible from "devel" if you are modifying the Ansible engine, or trying out
features under development. This is a rapidly changing source of code and can
become unstable at any point.
INFO default scenario test matrix: dependency, create, prepare, converge
INFO Performing prerun with role_name_check=0...
INFO Set ANSIBLE_LIBRARY=/Users/jackzhang/.cache/ansible-compat/81a7e6/modules:/Users/jackzhang/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
INFO Set ANSIBLE_COLLECTIONS_PATH=/Users/jackzhang/.cache/ansible-compat/81a7e6/collections:/Users/jackzhang/.ansible/collections:/usr/share/ansible/collections
INFO Set ANSIBLE_ROLES_PATH=/Users/jackzhang/.cache/ansible-compat/81a7e6/roles:/Users/jackzhang/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
INFO Using /Users/jackzhang/.cache/ansible-compat/81a7e6/roles/acme.my_new_role_docker symlink to current repository in order to enable Ansible to find the role using its expected full name.
INFO Running default > dependency
WARNING Skipping, missing the requirements file.
WARNING Skipping, missing the requirements file.
INFO Running default > create
WARNING Skipping, instances already created.
INFO Running default > prepare
WARNING Skipping, prepare playbook not configured.
INFO Running default > converge
INFO Sanity checks: 'docker'
[WARNING]: You are running the development version of Ansible. You should only
run Ansible from "devel" if you are modifying the Ansible engine, or trying out
features under development. This is a rapidly changing source of code and can
become unstable at any point.
PLAY [Converge] ****************************************************************
TASK [Gathering Facts] *********************************************************
ok: [instance]
TASK [Include acme.my_new_role_docker] *****************************************
PLAY [Converge] ****************************************************************
TASK [Gathering Facts] *********************************************************
ok: [instance]
TASK [Include acme.my_new_role_docker] *****************************************
ok: [instance] => {
"msg": "/Users/jackzhang/.cache/molecule/my_new_role_docker/default"
}
PLAY RECAP *********************************************************************
instance : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0timblaktu commented
Thanks @zhan9san, I will fix my playbook syntax as you've noted. However I can't imagine that this would be the cause of the error I reported. It appears that molecule isn't even able to find and/or execute ansible in the virtual python environment I'm using. This is the same virtual environment that molecule is installed into and running in.
timblaktu commented
@zhan9san changing my deb10-headless molecule scenario's converge.yml playbook to correctly invoke the desired playbook:
---
- name: Import Main Playbook to be Tested
import_playbook: ../../tblack-deb10-headless.ymlhas no effect on the reported error.
timblaktu commented
News flash
It just occurred to me that this error means that ansible is not finding python in the target container.
I then removed ANSIBLE_PYTHON_INTERPRETER from my environment, and the started focusing on why ansible is not able to discover python at /usr/bin/python in the container. This was cleared up pretty quickly when I shelled into the container and saw no python.
So, I have confirmed this was cockpit error on my part, and confirmed it's discovering python interpreter after changing the docker driver config in molecule.yml to use the python:buster image. I can now see that there are going to be other pre-reqs that my playbooks will require, and I can use a prepare step to ensure I satisfy these.
UPDATE: I eventually came to learn, and not from reading the molecule docs, that there is a Dockerfile.j2 template placed in the scenario dir when you initialize it using docker driver type. I had initially initialized my scenario using default driver, and manually changed the driver in the molecule.yml, so I had no idea where the Dockerfile.j2 came from until I found it in a search of the molecule repo. The only mention of this template is on this page which discussed how to modify the template, but nothing about where it comes from.