nodes not ready due to cni uninitialized with 'openshift_use_flannel=true'

Question

nodes not ready due to cni uninitialized with 'openshift_use_flannel=true'

Closed this issue 4 years ago · 6 comments

yu2003w commented 5 years ago

Description

On a all-in-one install, node did not become ready because cni uninitialized.
flannel is used instead of open shift-sdn.

Version

ansible 2.6.19
  config file = /usr/share/ansible/openshift-ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Jun 11 2019, 14:33:56) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

Steps To Reproduce

run prerequisite
ansible-playbook -i ./hosts playbooks/prerequisites.yml

PLAY RECAP *********************************************************************************************************************************
172.16.211.60              : ok=69   changed=9    unreachable=0    failed=0   
localhost                  : ok=11   changed=0    unreachable=0    failed=0   


INSTALLER STATUS ***************************************************************************************************************************
Initialization  : Complete (0:02:07)
Tuesday 08 October 2019  01:43:35 -0700 (0:00:00.042)       0:04:00.426 ******* 
=============================================================================== 
Ensure openshift-ansible installer package deps are installed --------------------------------------------------------------------- 101.55s
openshift_excluder : Install docker excluder - yum --------------------------------------------------------------------------------- 60.64s
container_runtime : Install Docker ------------------------------------------------------------------------------------------------- 15.90s
openshift_repos : Ensure libselinux-python is installed ----------------------------------------------------------------------------- 8.18s
os_firewall : Install firewalld packages -------------------------------------------------------------------------------------------- 7.99s
container_runtime : Create credentials for oreg_url --------------------------------------------------------------------------------- 6.33s
openshift_repos : refresh cache ----------------------------------------------------------------------------------------------------- 3.93s
container_runtime : Get current installed Docker version ---------------------------------------------------------------------------- 3.36s
container_runtime : Fixup SELinux permissions for docker ---------------------------------------------------------------------------- 3.10s
Gathering Facts --------------------------------------------------------------------------------------------------------------------- 1.57s
os_firewall : Ensure iptables services are not enabled ------------------------------------------------------------------------------ 1.22s
Gather Cluster facts ---------------------------------------------------------------------------------------------------------------- 0.80s
container_runtime : Update registries.conf ------------------------------------------------------------------------------------------ 0.70s
Initialize openshift.node.sdn_mtu --------------------------------------------------------------------------------------------------- 0.61s
container_runtime : Start the Docker service ---------------------------------------------------------------------------------------- 0.58s
container_runtime : Set various Docker options -------------------------------------------------------------------------------------- 0.56s
get openshift_current_version ------------------------------------------------------------------------------------------------------- 0.52s
os_firewall : Start and enable firewalld service ------------------------------------------------------------------------------------ 0.48s
Detecting Operating System from ostree_booted --------------------------------------------------------------------------------------- 0.47s
openshift_repos : Remove openshift_additional.repo file ----------------------------------------------------------------------------- 0.44s

run deploy
ansible-playbook -i ./hosts playbooks/deploy_cluster.yml

TASK [openshift_web_console : debug] *******************************************************************************************************
Tuesday 08 October 2019  02:33:55 -0700 (0:00:00.802)       0:35:24.910 ******* 
ok: [172.16.211.60] => {
    "msg": []
}

TASK [openshift_web_console : Report console errors] ***************************************************************************************
Tuesday 08 October 2019  02:33:55 -0700 (0:00:00.318)       0:35:25.229 ******* 
fatal: [172.16.211.60]: FAILED! => {"changed": false, "msg": "Console install failed."}

PLAY RECAP *********************************************************************************************************************************
172.16.211.60              : ok=562  changed=228  unreachable=0    failed=1   
localhost                  : ok=11   changed=0    unreachable=0    failed=0   


INSTALLER STATUS ***************************************************************************************************************************
Initialization              : Complete (0:00:14)
Health Check                : Complete (0:00:49)
Node Bootstrap Preparation  : Complete (0:07:05)
etcd Install                : Complete (0:00:53)
Master Install              : Complete (0:07:09)
Master Additional Install   : Complete (0:07:08)
Node Join                   : Complete (0:00:15)
Hosted Install              : Complete (0:00:49)
Web Console Install         : In Progress (0:10:44)
	This phase can be restarted by running: playbooks/openshift-web-console/config.yml
Tuesday 08 October 2019  02:33:55 -0700 (0:00:00.277)       0:35:25.507 ******* 
=============================================================================== 
openshift_web_console : Verify that the console is running ------------------------------------------------------------------------ 631.82s
cockpit : Install cockpit-ws ------------------------------------------------------------------------------------------------------ 386.70s
openshift_node : Install node, clients, and conntrack packages -------------------------------------------------------------------- 141.43s
openshift_ca : Install the base package for admin tooling -------------------------------------------------------------------------- 78.67s
openshift_control_plane : Wait for all control plane pods to become ready ---------------------------------------------------------- 78.46s
openshift_node : install needed rpm(s) --------------------------------------------------------------------------------------------- 67.70s
openshift_excluder : Install openshift excluder - yum ------------------------------------------------------------------------------ 63.52s
Run health checks (install) - EL --------------------------------------------------------------------------------------------------- 49.54s
openshift_control_plane : Wait for control plane pods to appear -------------------------------------------------------------------- 46.56s
openshift_cli : Install clients ---------------------------------------------------------------------------------------------------- 36.22s
openshift_excluder : Install docker excluder - yum --------------------------------------------------------------------------------- 28.11s
openshift_node : Install GlusterFS storage plugin dependencies --------------------------------------------------------------------- 24.34s
openshift_node_group : Wait for the sync daemonset to become ready and available --------------------------------------------------- 11.25s
openshift_node : Install iSCSI storage plugin dependencies -------------------------------------------------------------------------- 9.94s
openshift_hosted : Create OpenShift router ------------------------------------------------------------------------------------------ 8.44s
openshift_node : Install dnsmasq ---------------------------------------------------------------------------------------------------- 8.18s
openshift_manageiq : Configure role/user permissions -------------------------------------------------------------------------------- 7.82s
etcd : Install openssl -------------------------------------------------------------------------------------------------------------- 7.72s
nickhammond.logrotate : nickhammond.logrotate | Install logrotate ------------------------------------------------------------------- 7.70s
openshift_cli : Install bash completion for oc tools -------------------------------------------------------------------------------- 7.68s


Failure summary:


  1. Hosts:    172.16.211.60
     Play:     Web Console
     Task:     Report console errors
     Message:  Console install failed.

Expected Results

installation should be successfully.

Observed Results

oc describe nodes

PIDPressure      False   Tue, 08 Oct 2019 02:35:53 -0700   Tue, 08 Oct 2019 02:11:27 -0700   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Tue, 08 Oct 2019 02:35:53 -0700   Tue, 08 Oct 2019 02:11:27 -0700   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Because node is not ready, pods couldn't be scheduled. So installation failed.

For long output or logs, consider using a gist

Additional Information

os version

Red Hat Enterprise Linux Server release 7.7 (Maipo)

inventory file

# Create an OSEv3 group that contains the masters, nodes, and etcd groups
[OSEv3:children]
masters
nodes
etcd

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
# SSH user, this user should allow ssh based auth without requiring a password
ansible_ssh_user=root

# If ansible_ssh_user is not root, ansible_become must be set to true
#ansible_become=true

openshift_deployment_type=openshift-enterprise
openshift_use_flannel=true
openshift_use_openshift_sdn=false
flannel_interface=eth0
openshift_disable_check=memory_availability,disk_availability
openshift_master_cluster_public_hostname=buzz1.center.com
openshift_master_cluster_hostname=buzz1.center.com
os_firewall_use_firewalld=true
openshift_cluster_monitoring_operator_install=false
#os_sdn_network_plugin_name=cni

# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider
#openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]

# host group for masters
[masters]
172.16.211.60

# host group for etcd
[etcd]
172.16.211.60

# host group for nodes, includes region info
[nodes]
172.16.211.60 openshift_public_ip=9.30.119.88 openshift_public_hostname=buzz1.center.com openshift_node_group_name='node-config-all-in-one'

Answer 1 · 2019-10-09T02:24:16.000Z

I also tried with calico as network plugin, it worked well.
So I guessed that flannel and default openshift-sdn are not supported any more.

Answer 2 · 2019-11-01T14:51:19.000Z

I've had a similar issue, you couldn't get into install with the default sdn plugin either?

Answer 3 · 2020-06-03T10:23:06.000Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

Answer 4 · 2020-07-03T12:16:53.000Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

Answer 5 · 2020-08-02T14:03:38.000Z

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Answer 6 · 2020-08-02T14:03:51.000Z

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.