Internal docker registry NFS volume creation fail with "Required value: must specify a volume type"

Question

Internal docker registry NFS volume creation fail with "Required value: must specify a volume type"

Closed this issue 5 years ago · 22 comments

Description

Unable to deploy OpenShift 3.11 clusters with docker registry backed by NFS storage, broken persistent-volumes.yml content.

Version

Please put the following version information in the code block
indicated below.

Your ansible version per ansible --version

ansible 2.8.1
  config file = None
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Apr  9 2019, 14:30:50) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

For installation was used last changes from release-3.11 branch

The output of git describe

openshift-ansible-3.11.123-1-8-ga03b285

Steps To Reproduce

Create inventory file with NFS storage for docker registry:

openshift_hosted_registry_storage_kind=nfs
openshift_hosted_registry_storage_access_modes=['ReadWriteMany']
openshift_hosted_registry_storage_nfs_directory=/exports
openshift_hosted_registry_storage_nfs_options='*(rw,root_squash)'
openshift_hosted_registry_storage_volume_name=registry
openshift_hosted_registry_storage_volume_size=100Gi

Run preparation and install playbooks

Expected Results

Successful OpenShift installation

Observed Results

Installation fails with error:

The PersistentVolume \"registry-volume\" is invalid: spec: Required value: must specify a volume type

fatal: [denis-master-1]: FAILED! => {
    "changed": false, 
    "cmd": [
        "oc", 
        "create", 
        "-f", 
        "/tmp/openshift-ansible-3vGorUB/persistent-volumes.yml", 
        "--config=/tmp/openshift-ansible-3vGorUB/admin.kubeconfig"
    ], 
    "delta": "0:00:00.207853", 
    "end": "2019-06-27 08:11:51.219853", 
    "failed_when_result": true, 
    "invocation": {
        "module_args": {
            "_raw_params": "oc create -f /tmp/openshift-ansible-3vGorUB/persistent-volumes.yml --config=/tmp/openshift-ansible-3vGorUB/admin.kubeconfig\n", 
            "_uses_shell": false, 
            "argv": null, 
            "chdir": null, 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "stdin": null, 
            "stdin_add_newline": true, 
            "strip_empty_ends": true, 
            "warn": true
        }
    }, 
    "msg": "non-zero return code", 
    "rc": 1, 
    "start": "2019-06-27 08:11:51.012000", 
    "stderr": "The PersistentVolume \"registry-volume\" is invalid: spec: Required value: must specify a volume type", 
    "stderr_lines": [
        "The PersistentVolume \"registry-volume\" is invalid: spec: Required value: must specify a volume type"
    ], 
    "stdout": "", 
    "stdout_lines": []
}

Persistent volumes file:

cat /tmp/openshift-ansible-3vGorUB/persistent-volumes.yml 
---
apiVersion: v1
kind: List
items:
- apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: "registry-volume"
    labels:
  spec:
    capacity:
      storage: "100Gi"
    accessModes: 
    - ReadWriteMany
    claimName: 
      registry-claim
      ...
    claimRef:
      name: registry-claim
      namespace: default

looks like file content is broken

Additional Information

If I remove keys openshift_hosted_registry_storage* from inventory file - installation pass.

Retried installation with openshift_enable_unsupported_configurations=True - same failure message.

OS on nodes: CentOS Linux release 7.6.1810 (Core)
Openshift ansible git head hash: a03b28590853da28d1edacb8183c724c943b1acf

Answer 1 · 2019-07-10T12:06:01.000Z

Same issue here installing OKD 3.11

Answer 2 · 2019-07-14T15:06:54.000Z

Same problem here. Using latest release-3.11 branch. Got error

TASK [openshift_persistent_volumes : Create PersistentVolumes] *****************************************************************************************************
Sunday 14 July 2019  08:00:33 -0700 (0:00:01.002)       0:00:23.333 ***********
fatal: [10.51.4.187]: FAILED! => {"changed": false, "cmd": ["oc", "create", "-f", "/tmp/openshift-ansible-Iol6cCt/persistent-volumes.yml", "--config=/tmp/openshift-ansible-Iol6cCt/admin.kubeconfig"], "delta": "0:00:00.305053", "end": "2019-07-14 08:00:33.900001", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2019-07-14 08:00:33.594948", "stderr": "The PersistentVolume \"registry-volume\" is invalid: spec: Required value: must specify a volume type", "stderr_lines": ["The PersistentVolume \"registry-volume\" is invalid: spec: Required value: must specify a volume type"], "stdout": "", "stdout_lines": []}

Content of PV yaml:

[root@tremolo1 openshift-ansible]# cat /tmp/openshift-ansible-Iol6cCt/persistent-volumes.yml
---
apiVersion: v1
kind: List
items:
- apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: "registry-volume"
    labels:
  spec:
    capacity:
      storage: "50Gi"
    accessModes:
    - ReadWriteMany
    claimName:
      registry-claim
      ...
    claimRef:
      name: registry-claim
      namespace: default

Answer 3 · 2019-07-17T19:27:19.000Z

We ran into the same issue. First thing I noticed was that Ansible had been updated to 2.8.1 as well which struck me as not an ideal situation.

We downgraded Ansible to 2.6.14 and also downgraded to openshift-ansible-3.11.104 which resolved the issue for now.

We aren't sure which on those are causing the issues but I have never had much success with a version of ansible that wasn't blessed for OpenShift operations.

Answer 4 · 2019-07-26T19:55:22.000Z

We ran into the same issue. First thing I noticed was that Ansible had been updated to 2.8.1 as well which struck me as not an ideal situation.

We downgraded Ansible to 2.6.14 and also downgraded to openshift-ansible-3.11.104 which resolved the issue for now.

We aren't sure which on those are causing the issues but I have never had much success with a version of ansible that wasn't blessed for OpenShift operations.

Thanks for the suggestion. Bumped into this issue as well.
Installation with latest openshift-ansible 3.11 (okd) worked with ansible 2.6.18 (pip) but not version 2.8.2 (EPEL).

Answer 5 · 2019-09-02T22:22:06.000Z

I've run into the same issue.

The PersistentVolume "registry-volume" is invalid: spec: Required value: must specify a volume type"

ansible 2.8.0
config file = None
configured module search path = [u'/Users/mcauleym/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /Library/Python/2.7/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 2.7.10 (default, Feb 22 2019, 21:55:15) [GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)]

How are are the folks using ansible 2.6.14? I get explicit messages when running the playbook that ansible 2.8+ is required?

We ran into the same issue. First thing I noticed was that Ansible had been updated to 2.8.1 as well which struck me as not an ideal situation.
We downgraded Ansible to 2.6.14 and also downgraded to openshift-ansible-3.11.104 which resolved the issue for now.
We aren't sure which on those are causing the issues but I have never had much success with a version of ansible that wasn't blessed for OpenShift operations.

Thanks for the suggestion. Bumped into this issue as well.
Installation with latest openshift-ansible 3.11 (okd) worked with ansible 2.6.18 (pip) but not version 2.8.2 (EPEL).

Answer 6 · 2019-09-02T22:26:03.000Z

Which playbook tells you you need to use 2.8+?

Pretty sure that isn't in any of the openshift-ansible playbooks.

Answer 7 · 2019-09-02T22:53:18.000Z

Which playbook tells you you need to use 2.8+?

Pretty sure that isn't in any of the openshift-ansible playbooks.

https://github.com/openshift/openshift-ansible/blob/release-3.11/requirements.txt
shows ansible=2.8.1. I thought I had a full stop error saying I needed 2.8+

I'll restore my VM's to their pre-install state and run again to provide a clean test. I am wondering my ansible isn't working properly. Will report back...

EDIT

ansible 2.6.14 doesn't work
mini2:openshift-ansible mcauleym$ ansible-playbook -i ../openshift_inventory playbooks/prerequisites.yml -vvv
ansible-playbook 2.6.12
config file = /Users/mcauleym/Documents/openshift/openshift-ansible/ansible.cfg
configured module search path = [u'/Users/mcauleym/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /Library/Python/2.7/site-packages/ansible
executable location = /usr/local/bin/ansible-playbook
python version = 2.7.10 (default, Feb 22 2019, 21:55:15) [GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)]
Using /Users/mcauleym/Documents/openshift/openshift-ansible/ansible.cfg as config file
Parsed /Users/mcauleym/Documents/openshift/openshift_inventory inventory source with ini plugin
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/rhel_subscribe/tasks/satellite.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/os_firewall/tasks/firewalld.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/os_firewall/tasks/iptables.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/openshift_excluder/tasks/install.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/openshift_excluder/tasks/exclude.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/container_runtime/tasks/common/pre.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/container_runtime/tasks/common/udev_workaround.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/container_runtime/tasks/common/atomic_proxy.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/container_runtime/tasks/docker_sanity.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/container_runtime/tasks/common/post.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/container_runtime/tasks/registry_auth.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/container_runtime/tasks/common/setup_docker_symlink.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/container_runtime/tasks/common/pre.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/container_runtime/tasks/common/udev_workaround.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/container_runtime/tasks/crio_firewall.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/container_runtime/tasks/common/post.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/container_runtime/tasks/registry_auth.yml
statically imported: /Users/mcauleym/Documents/openshift/openshift-ansible/roles/container_runtime/tasks/common/setup_docker_symlink.yml
ERROR! Unexpected Exception, this is probably a bug: 'module' object has no attribute '_vendor'
the full traceback was:

Traceback (most recent call last):
File "/usr/local/bin/ansible-playbook", line 118, in
exit_code = cli.run()
File "/Library/Python/2.7/site-packages/ansible/cli/playbook.py", line 122, in run
results = pbex.run()
File "/Library/Python/2.7/site-packages/ansible/executor/playbook_executor.py", line 89, in run
self._tqm.load_callbacks()
File "/Library/Python/2.7/site-packages/ansible/executor/task_queue_manager.py", line 193, in load_callbacks
for callback_plugin in callback_loader.all(class_only=True):
File "/Library/Python/2.7/site-packages/ansible/plugins/loader.py", line 489, in all
module = self._load_module_source(name, path)
File "/Library/Python/2.7/site-packages/ansible/plugins/loader.py", line 357, in _load_module_source
module = imp.load_source(full_name, path, module_file)
File "/Users/mcauleym/Documents/openshift/openshift-ansible/roles/lib_utils/callback_plugins/aa_version_requirement.py", line 10, in
from pkg_resources import parse_version
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/init.py", line 90, in
packaging = pkg_resources._vendor.packaging
AttributeError: 'module' object has no attribute '_vendor'

How are you running ansible 2.6.14? What OS?

Answer 8 · 2019-09-03T01:01:44.000Z

I had it backwards. I checkout 3.10, and run:
mini2:openshift-ansible mcauleym$ ansible-playbook -i ../openshift_inventory playbooks/prerequisites.yml
FATAL: Current Ansible version (2.8.2) is not supported. Supported versions: 2.4.3.0 or newer but less than 2.8.0

either way - I can't get ansible 2.6, 2.7 to run on OSX or CentOS 7. Installed via pip. Going to install Cent 6.5 and try from there.

Answer 9 · 2019-09-03T03:47:26.000Z

I ran into the same error with OpenShit 3.11 and Ansible 2.8.2

Answer 10 · 2019-09-03T04:31:21.000Z

The Jinja file persistent-volume.yml.j2 actually does not has a type section

it missing volumeMode: Filesystem
it missing items like this

  mountOptions:
  - vers=4.1
  nfs:
    path: /export/pvc-984b4aca-ca06-11e9-97ae-00000a15074b
    server: 10.0.0.3

persistent-volume.yml.j2

{% for volume in persistent_volumes %}
- apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: "{{ volume.name }}"
{% if volume.annotations is defined %}
    annotations:
{% for annotation in volume.annotations %}
      {{ annotation }}
{% endfor %}
{% endif %}
{% if volume.labels is defined and volume.labels is mapping %}
    labels:
{% for key,value in volume.labels.items() %}
      {{ key }}: {{ value }}
{% endfor %}
{% endif %}
  spec:
    capacity:
      storage: "{{ volume.capacity }}"
    accessModes: {{ volume.access_modes | lib_utils_to_padded_yaml(2, 2) }}
    {{ (volume.storage.keys() | list)[0] }}: {{ volume.storage[(volume.storage.keys() | list)[0]] | lib_utils_to_padded_yaml(3, 2) }}
{% if 'claimName' in volume.storage %}
    claimRef:
      name: {{ volume.storage.claimName }}
      namespace: default
{% endif %}
{% endfor %}

Answer 11 · 2019-09-03T04:48:35.000Z

I had it backwards. I checkout 3.10, and run:
mini2:openshift-ansible mcauleym$ ansible-playbook -i ../openshift_inventory playbooks/prerequisites.yml
FATAL: Current Ansible version (2.8.2) is not supported. Supported versions: 2.4.3.0 or newer but less than 2.8.0

either way - I can't get ansible 2.6, 2.7 to run on OSX or CentOS 7. Installed via pip. Going to install Cent 6.5 and try from there.

CentOS 7 works fine for me.

# install dependencies 
sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum -y install python-pip git httpd-tools java-1.8.0-openjdk-headless patch python-passlib
# install compatible ansible version for openshift-ansible
sudo pip install ansible==2.6.18
sudo pip install --upgrade setuptools

Answer 12 · 2020-01-07T11:27:06.000Z

I am having the same issue with the latest 3.11 version. Any suggestions for this problem?

Answer 13 · 2020-01-07T17:08:09.000Z

hi,
use ansible with version less than 2.8.0

Answer 14 · 2020-01-08T09:12:52.000Z

Installing Ansible 2.6.18 like @pekare above was giving errors. I was able however to get 2.7.10 working. This solved my problem.

sudo pip install ansible==2.7.10 --user
sudo pip install --upgrade setuptools --user

Answer 15 · 2020-03-30T16:07:38.000Z

Looks like it is injecting "spec.claimName", but that isn't valid according to these docs:
https://docs.openshift.com/container-platform/3.11/rest_api/api/v1.PersistentVolume.html

Answer 16 · 2020-03-30T16:15:02.000Z

Also, seems like this portion of the template isn't putting out any data or is responsible for digging the claimName out somehow
{{ (volume.storage.keys() | list)[0] }}: {{ volume.storage[(volume.storage.keys() | list)[0]] | lib_utils_to_padded_yaml(3, 2) }}

Answer 17 · 2020-03-30T18:01:57.000Z

I see @abutcher 's name on the README. I wonder if he's had the chance to review this and could point in some direction where the bad data is coming from.

In the mean time, anyone have a workaround besides changing Ansible versions? That's not a trivial change in my environment.

Answer 18 · 2020-03-30T21:41:39.000Z

I found a workaround setting several variables openshift_*_storage_kind to nfs. Also had to set openshift_enable_unsupported_configurations: True because nfs isn't supported for logging at least.

openshift_hosted_registry_storage_kind: nfs
openshift_metrics_storage_kind: nfs
openshift_logging_storage_kind: nfs
openshift_loggingops_storages_kind: nfs
openshift_hosted_etcd_storage_kind: nfs

Answer 19 · 2020-04-16T01:01:35.000Z

We ran into the same issue. First thing I noticed was that Ansible had been updated to 2.8.1 as well which struck me as not an ideal situation.

We downgraded Ansible to 2.6.14 and also downgraded to openshift-ansible-3.11.104 which resolved the issue for now.

We aren't sure which on those are causing the issues but I have never had much success with a version of ansible that wasn't blessed for OpenShift operations.

I am running into this issue too. Ansible 2.8.8.
Red Hat has blessed the newer versions of Ansible. As of when we deployed here it was 2.8, now it's 2.9:

(from https://docs.openshift.com/container-platform/3.11/install/host_preparation.html)

Answer 20 · 2020-04-27T22:38:30.000Z

Ran into this with:

openshift-ansible 3.11.170 2.git.5.8802564.el7
ansible 2.9.5

Downgraded to ansible 2.7.10 and got past it.
Thanks @denis256

Answer 21 · 2020-05-28T07:08:44.000Z

have a try,
docker pull ho11223344/origin-ansible-eazytec:v3.11
from openshift/origin-ansible:v3.11, ansible version : 2.6.5

Answer 22 · 2020-05-31T08:52:19.000Z

Closing issue, fix is to use ansible lower than 2.8.0