Failed TASK [stackhpc.libvirt-vm : Ensure the VM volumes exist]
mleone87 opened this issue · 12 comments
mleone87 commented
Hi, i'm following the tutorial but the ./dev/seed-deploy.sh phase fails with this error(sorry for italian error in ansible)
the host is a bare metal 4C 32GB Centos7 latest install
TASK [stackhpc.libvirt-vm : Ensure the VM volumes exist] ********************************************************************************************************
failed: [seed-hypervisor] (item={u'image': u'https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2', u'capacity': u'50G', u'name': u'seed-root', u'pool': u'default', u'format': u'qcow2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"capacity": "50G", "format": "qcow2", "image": "https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2", "name": "seed-root", "pool": "default"}, "msg": "non-zero return code", "rc": 1, "stderr": "Shared connection to 192.168.33.4 closed.\r\n", "stderr_lines": ["Shared connection to 192.168.33.4 closed."], "stdout": "Unexpected error while getting volume info\r\nerrore: impossibile ottenere il vol 'seed-root'\r\nerrore: Volume di storage non trovato: nessun vol di storage con percorso corrispondente a 'seed-root'\r\n", "stdout_lines": ["Unexpected error while getting volume info", "errore: impossibile ottenere il vol 'seed-root'", "errore: Volume di storage non trovato: nessun vol di storage con percorso corrispondente a 'seed-root'"]}
failed: [seed-hypervisor] (item={u'capacity': u'100G', u'name': u'seed-data', u'pool': u'default', u'format': u'qcow2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"capacity": "100G", "format": "qcow2", "name": "seed-data", "pool": "default"}, "msg": "non-zero return code", "rc": 1, "stderr": "Shared connection to 192.168.33.4 closed.\r\n", "stderr_lines": ["Shared connection to 192.168.33.4 closed."], "stdout": "Unexpected error while getting volume info\r\nerrore: impossibile ottenere il vol 'seed-data'\r\nerrore: Volume di storage non trovato: nessun vol di storage con percorso corrispondente a 'seed-data'\r\n", "stdout_lines": ["Unexpected error while getting volume info", "errore: impossibile ottenere il vol 'seed-data'", "errore: Volume di storage non trovato: nessun vol di storage con percorso corrispondente a 'seed-data'"]}
failed: [seed-hypervisor] (item={u'capacity': u'389120', u'name': u'seed-configdrive', u'format': u'raw', u'device': u'cdrom', u'image': u'/opt/kayobe/images/seed.iso', u'pool': u'default'}) => {"ansible_loop_var": "item", "changed": false, "item": {"capacity": "389120", "device": "cdrom", "format": "raw", "image": "/opt/kayobe/images/seed.iso", "name": "seed-configdrive", "pool": "default"}, "msg": "non-zero return code", "rc": 1, "stderr": "Shared connection to 192.168.33.4 closed.\r\n", "stderr_lines": ["Shared connection to 192.168.33.4 closed."], "stdout": "Unexpected error while getting volume info\r\nerrore: impossibile ottenere il vol 'seed-configdrive'\r\nerrore: Volume di storage non trovato: nessun vol di storage con percorso corrispondente a 'seed-configdrive'\r\n", "stdout_lines": ["Unexpected error while getting volume info", "errore: impossibile ottenere il vol 'seed-configdrive'", "errore: Volume di storage non trovato: nessun vol di storage con percorso corrispondente a 'seed-configdrive'"]}markgoddard commented
Hi @mleone87, thanks for raising this issue. I translated the error message:
error: unable to get the 'seed-root' vol error: storage volume not found: no storage vol with path corresponding to 'seed-root'
I think this may be the cause of the problem:
output=$(virsh vol-info --pool "$POOL" --vol "$NAME" 2>&1)
result=$?
if [[ $result -eq 0 ]]; then
echo '{"changed": false}'
exit 0
elif ! echo "$output" | grep 'Storage volume not found' >/dev/null 2>&1; then
That code is in the stackhpc.libvirt-vm role, in the file files/virt_volume.sh. We might need to try something like this:
- output=$(virsh vol-info --pool "$POOL" --vol "$NAME" 2>&1)
+ output=$(LANG=C virsh vol-info --pool "$POOL" --vol "$NAME" 2>&1)
Is that something you could try?
markgoddard commented
I've tried experimenting with locales, and it's difficult to change the language of the response because the libvirt daemon process still keeps the original locale. I think we would need to avoid relying on English in stdout.
In the meantime, could you try again with your language set to English?
mleone87 commented
setting locale to eng fixes this task fail even if the virsh output is still in italian if you do it manually:
sudo virsh vol-list --pool default
Nome Percorso
------------------------------------------------------------------------------
seed-configdrive /var/lib/libvirt/images/seed-configdrive
seed-data /var/lib/libvirt/images/seed-data
seed-root /var/lib/libvirt/images/seed-rootsudo virsh list --all
Id Nome Stato
----------------------------------------------------
1 seed in esecuzione
(now it fails there)
PLAY [Ensure known hosts are configured] ****************************************************************************************************************
TASK [ssh-known-host : Scan for SSH keys] ***************************************************************************************************************
ok: [seed -> localhost] => (item=192.168.33.5)
TASK [ssh-known-host : Ensure SSH keys are in known hosts] **********************************************************************************************
changed: [seed -> localhost] => (item=[{'stderr_lines': [u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4'], 'ansible_loop_var': u'item', u'end': u'2020-02-06 12:06:51.511766', u'stderr': u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4\n# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4\n# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'stdout': u'192.168.33.5 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCRUGmfkL+0wj7jH5WzZlbCoTFiB7ZFilsyRAsS7c3X5h1wd8tXTclgVw3WWztwYXPZYBuBGp9+2I0qiFQnMZeM=\n192.168.33.5 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIjVPQZvkMCKlEpYJ96L7iAlAWj/8UqFTfuiTNXctfJfz+63AcJmNjTWOXi5kAUtOKW+qwcPutqK2ytAOLeUih8GU6mt2V1niVbSPGpVn2qgDe5Uv5ZVPdoOw0iAekOfJL39jmR9neDYt1UvWMfBBnBy4vYjMyuUFvVd+jMJRISNDxd/oYUUWboTN526oLzXSRRx2Arps66rCa5vfk7sTfld3MOjHjuo0slAS1RSe2eQdo5tMenRSo2CRAZrg00PZcIVIYUKfb9A7vJ8XA74+4tLftLnwTXSLyvzl1dYpvDCgZGFooIxUhDZNNyBZ3GppvZ1JEktClPxJj8Mlr+13H\n192.168.33.5 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID823ZE52p6QGarb7noS4LeLvePKZOWOpicET/NH9up+', u'changed': False, 'ansible_facts': {u'discovered_interpreter_python': u'/usr/bin/python'}, 'failed': False, u'delta': u'0:00:00.180425', u'cmd': [u'ssh-keyscan', u'192.168.33.5'], 'item': u'192.168.33.5', u'rc': 0, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': False, u'strip_empty_ends': True, u'_raw_params': u'ssh-keyscan 192.168.33.5', u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin_add_newline': True, u'stdin': None}}, u'start': u'2020-02-06 12:06:51.331341'}, u'192.168.33.5 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCRUGmfkL+0wj7jH5WzZlbCoTFiB7ZFilsyRAsS7c3X5h1wd8tXTclgVw3WWztwYXPZYBuBGp9+2I0qiFQnMZeM='])
changed: [seed -> localhost] => (item=[{'stderr_lines': [u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4'], 'ansible_loop_var': u'item', u'end': u'2020-02-06 12:06:51.511766', u'stderr': u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4\n# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4\n# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'stdout': u'192.168.33.5 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCRUGmfkL+0wj7jH5WzZlbCoTFiB7ZFilsyRAsS7c3X5h1wd8tXTclgVw3WWztwYXPZYBuBGp9+2I0qiFQnMZeM=\n192.168.33.5 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIjVPQZvkMCKlEpYJ96L7iAlAWj/8UqFTfuiTNXctfJfz+63AcJmNjTWOXi5kAUtOKW+qwcPutqK2ytAOLeUih8GU6mt2V1niVbSPGpVn2qgDe5Uv5ZVPdoOw0iAekOfJL39jmR9neDYt1UvWMfBBnBy4vYjMyuUFvVd+jMJRISNDxd/oYUUWboTN526oLzXSRRx2Arps66rCa5vfk7sTfld3MOjHjuo0slAS1RSe2eQdo5tMenRSo2CRAZrg00PZcIVIYUKfb9A7vJ8XA74+4tLftLnwTXSLyvzl1dYpvDCgZGFooIxUhDZNNyBZ3GppvZ1JEktClPxJj8Mlr+13H\n192.168.33.5 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID823ZE52p6QGarb7noS4LeLvePKZOWOpicET/NH9up+', u'changed': False, 'ansible_facts': {u'discovered_interpreter_python': u'/usr/bin/python'}, 'failed': False, u'delta': u'0:00:00.180425', u'cmd': [u'ssh-keyscan', u'192.168.33.5'], 'item': u'192.168.33.5', u'rc': 0, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': False, u'strip_empty_ends': True, u'_raw_params': u'ssh-keyscan 192.168.33.5', u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin_add_newline': True, u'stdin': None}}, u'start': u'2020-02-06 12:06:51.331341'}, u'192.168.33.5 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIjVPQZvkMCKlEpYJ96L7iAlAWj/8UqFTfuiTNXctfJfz+63AcJmNjTWOXi5kAUtOKW+qwcPutqK2ytAOLeUih8GU6mt2V1niVbSPGpVn2qgDe5Uv5ZVPdoOw0iAekOfJL39jmR9neDYt1UvWMfBBnBy4vYjMyuUFvVd+jMJRISNDxd/oYUUWboTN526oLzXSRRx2Arps66rCa5vfk7sTfld3MOjHjuo0slAS1RSe2eQdo5tMenRSo2CRAZrg00PZcIVIYUKfb9A7vJ8XA74+4tLftLnwTXSLyvzl1dYpvDCgZGFooIxUhDZNNyBZ3GppvZ1JEktClPxJj8Mlr+13H'])
changed: [seed -> localhost] => (item=[{'stderr_lines': [u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4'], 'ansible_loop_var': u'item', u'end': u'2020-02-06 12:06:51.511766', u'stderr': u'# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4\n# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4\n# 192.168.33.5:22 SSH-2.0-OpenSSH_7.4', u'stdout': u'192.168.33.5 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCRUGmfkL+0wj7jH5WzZlbCoTFiB7ZFilsyRAsS7c3X5h1wd8tXTclgVw3WWztwYXPZYBuBGp9+2I0qiFQnMZeM=\n192.168.33.5 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIjVPQZvkMCKlEpYJ96L7iAlAWj/8UqFTfuiTNXctfJfz+63AcJmNjTWOXi5kAUtOKW+qwcPutqK2ytAOLeUih8GU6mt2V1niVbSPGpVn2qgDe5Uv5ZVPdoOw0iAekOfJL39jmR9neDYt1UvWMfBBnBy4vYjMyuUFvVd+jMJRISNDxd/oYUUWboTN526oLzXSRRx2Arps66rCa5vfk7sTfld3MOjHjuo0slAS1RSe2eQdo5tMenRSo2CRAZrg00PZcIVIYUKfb9A7vJ8XA74+4tLftLnwTXSLyvzl1dYpvDCgZGFooIxUhDZNNyBZ3GppvZ1JEktClPxJj8Mlr+13H\n192.168.33.5 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID823ZE52p6QGarb7noS4LeLvePKZOWOpicET/NH9up+', u'changed': False, 'ansible_facts': {u'discovered_interpreter_python': u'/usr/bin/python'}, 'failed': False, u'delta': u'0:00:00.180425', u'cmd': [u'ssh-keyscan', u'192.168.33.5'], 'item': u'192.168.33.5', u'rc': 0, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': False, u'strip_empty_ends': True, u'_raw_params': u'ssh-keyscan 192.168.33.5', u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin_add_newline': True, u'stdin': None}}, u'start': u'2020-02-06 12:06:51.331341'}, u'192.168.33.5 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID823ZE52p6QGarb7noS4LeLvePKZOWOpicET/NH9up+'])
PLAY RECAP **********************************************************************************************************************************************
seed : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
PLAY [Determine whether user bootstrapping is required] *************************************************************************************************
TASK [Check whether the host is accessible via SSH] *****************************************************************************************************
ok: [seed -> localhost]
TASK [Group hosts requiring kayobe user bootstrapping] **************************************************************************************************
ok: [seed]
TASK [Display a message when bootstrapping is required] *************************************************************************************************
ok: [seed] => {
"msg": "Cannot access host via SSH using Kayobe Ansible user account - attempting bootstrap\n"
}
PLAY [Ensure the Kayobe Ansible user account exists] ****************************************************************************************************
TASK [Gathering Facts] **********************************************************************************************************************************
ok: [seed]
TASK [singleplatform-eng.users : Creating groups] *******************************************************************************************************
TASK [singleplatform-eng.users : Per-user group creation] ***********************************************************************************************
fatal: [seed]: FAILED! => {"msg": "Timeout (12s) waiting for privilege escalation prompt: "}
But it could be a locale problem too, maybe it's worth to restart from scratch with a full english install?
markgoddard commented
I pushed a fix to the stackhpc.libvirt-vm role. It should work in Italian now. The above error looks like an ansible issue. Did you try multiple times?
mleone87 commented
yes, but the problem is persistent :)
markgoddard commented
You could at least try a reboot before reinstall.
mleone87 commented
I can confirm that the problem persist even with a fresh install on a full LANG=en_EN install... Maybe smth changed on kayobe scripts & ansible scripts?
markgoddard commented
Hmm. Can you get a simple ansible command to execute via ssh (not using local connection), also with become: true?
mleone87 commented
I can passwordless ssh into seed
$ ssh centos@192.168.33.5
Last login: Thu Feb 6 19:37:30 2020 from gatewayI can passwordless sudo seed
[centos@seed ~]$ sudo ls
[centos@seed ~]$I cant find a way to run ansible, the script ./dev/seed-deploy.sh develop too deep to understand what's going on.
[mario@kayobe kayobe]$ source /home/mario/kayobe-venv/bin/activate
(kayobe-venv) [mario@kayobe kayobe]$ source /home/mario/kayobe
(kayobe-venv) [mario@kayobe kayobe]$ source /home/mario/kayobe/config/src/kayobe-config/kayobe-env
Using Kayobe config from /home/mario/kayobe/config/src/kayobe-config
(kayobe-venv) [mario@kayobe kayobe]$ ansible -i /home/mario/kayobe/config/src/kayobe-config/etc/kayobe/inventory/hosts -m ping seed
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user
configurable on deprecation. This feature will be removed in version 2.10. Deprecation warnings can be disabled by setting deprecation_warnings=False in
ansible.cfg.
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
[WARNING]: Found both group and host with same name: seed
[WARNING]: Found both group and host with same name: seed-hypervisor
seed | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname seed: Name or service not known",
"unreachable": true
}
markgoddard commented
Ah, that will not work due to the lack of hostname resolution. We have group variables when running kayobe playbooks that set ansible_host appropriately but these won't be used for an ad-hoc command.
Are you able to use IRC? I'm mgoddard on #openstack-kolla.
markgoddard commented
I'm running through a-universe-from-nothing on a fresh install of CentOS 7 on bare metal and have passed the above point - my seed is now configured.
There is a 'cheat script' in a-universe-from-nothing: https://github.com/stackhpc/a-universe-from-nothing/blob/stable/train/a-universe-from-nothing.sh. You won't learn as much but it might help remove any copy/paste type errors.
Alternatively, if you just want to try out an all-in-one combined controller/compute node then there is less to go wrong and it is faster: https://docs.openstack.org/kayobe/latest/development/automated.html#overcloud. I would suggest using the stable/train branch for both kayobe and kayobe-config.
mleone87 commented
The problem were the default centos firewalld rules, disabling firewalld and using the configure-networking script will work!