openshift/openshift-ansible

cockpit/kubernetes-image no longer available from dockerhub

pgress opened this issue · 25 comments

pgress commented

Description

On a fresh cluster-installation the playbooks fail at the docker_image_availability-step. Reason for this issue is, that the cockpit/kubernetes:latest image is deleted. It looks like the work on this image ist discontinued.
See also cockpit-project/cockpit@37aa31e

Version
* Your ansible version per `ansible --version`
ansible 2.9.4
  config file = None
  configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.7.5 (default, Oct 17 2019, 12:25:15) [GCC 8.3.0]

* The output of `git describe`
openshift-ansible-3.11.174-1
Steps To Reproduce
  1. Just follow the instruction to install any sort of cluster
Expected Results

A working Cluster in initial state

Observed Results
     One or more checks failed
      check "docker_image_availability":
                One or more required container images are not available:
                    docker.io/cockpit/kubernetes:latest
                Checked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>
js-max commented

got errors also deploying 3.11 here, same msg

js-max commented

Checking the code if I'm wrong please correct me

defaults/main.yml file will have

openshift_health_check_required_images: "{{ [] + l_required_node_images + l_cockpit_images + l_master_required_images + l_etcd_required_images + l_atomic_node_required_images }}"

l_cockpit_images var is evaluated two lines before with a ternary

l_cockpit_images: "{{ (openshift_hosted_manage_registry_console | bool) | ternary([openshift_cockpit_deployer_image], []) }}"

so as a quick workaround openshift_hosted_manage_registry_console=false at inventory file is enough for a basic installation

different configs should need to be checked accordingly, tested and worked as expected on my side but with that approach openshift install will continue to check everything apart cockpit

really not aware that this should be handled like a var setup at inventory or the code should be changed

  • ❤️1
ZrytyADHD commented

I have the same error today installing cluster several times. I write on cockpit git and also try to pull image to my private repo from other clusterr.

timbrd commented

I've uploaded a recent cockpit/kubernetes image to my personal namespace on dockerhub.

With the following line in your inventory, you can deploy a cluster with a working registry-console:

openshift_cockpit_deployer_image='docker.io/timbordemann/cockpit-kubernetes:latest'

Btw: This seems to be the most recent source, if you want to build the image yourself: https://github.com/cockpit-project/cockpit/tree/195.6/containers/kubernetes

  • 👍9
  • 😄1
  • 🎉1
  • ❤️6
  • 👀1
thdevelthreesixtiy commented

Thx very much to timbrd for the copy.
You made my day!

aroute commented

Is there a way to fix this on an already-running OKD 3.11 cluster? My registry-console pod has ImagePullBackOff error due to this issue.

pgress commented

@aroute Yes, you can use the image from @timbrd and put it in your deployment for the registry-console

  • 👍1
aroute commented

okay, that worked. I built the image, which also worked. But, this must not be a permanent solution? Is anyone looking to resolve this so we don't have to apply our own work around?

timbrd commented

okay, that worked. I built the image, which also worked. But, this must not be a permanent solution? Is anyone looking to resolve this so we don't have to apply our own work around?

@ZrytyADHD has opened an issue on the cockpit project. Hopefully they reupload the image to the official namespace again soon.

magick93 commented

I have tried the following:

  • docker pull cockpit/kubernetes:latest
  • docker pull registry.access.redhat.com/openshift3/registry-console:latest
  • docker pull timbordemann/kubernetes:latest
  • docker pull openshift3/registry-console
  • oc import-image openshift3/registry-console --from=registry.access.redhat.com/openshift3/registry-console --confirm

None have worked.

I'm using Docker version 1.13.1, build cccb291/1.13.1 for Okd 3.11.

magick93 commented

This playbook will pull the image (tested on Centos 7)

---
- name: Fix missing registry console
  hosts: masters
  gather_facts: True

  tasks:

    - name: ensure /etc/rhsm/ca/redhat-uep.pem exists
      copy:
        content: ""
        dest: /etc/rhsm/ca/redhat-uep.pem
        owner: root
      become: yes


    - name: pull registry.access.redhat.com/openshift3/registry-console:v3.11.170-5
      docker_image:
        name: registry.access.redhat.com/openshift3/registry-console:v3.11.170-5
        source: pull
zilmarr commented

Thanks @timbrd ! I owe you a beer.

harrisiftikhar commented

I've uploaded a recent cockpit/kubernetes image to my personal namespace on dockerhub.

With the following line in your inventory, you can deploy a cluster with a working registry-console:

openshift_cockpit_deployer_image='docker.io/timbordemann/cockpit-kubernetes:latest'

Btw: This seems to be the most recent source, if you want to build the image yourself: https://github.com/cockpit-project/cockpit/tree/195.6/containers/kubernetes

Thanks for the response. I have added openshift_cockpit_deployer_image=docker.io/timbordemann/cockpit-kubernetes in /etc/ansible/hosts file
and I am able to pull the image
[root@master openshift-ansible]# docker pull docker.io/timbordemann/cockpit-kubernetes:latest
Trying to pull repository docker.io/timbordemann/cockpit-kubernetes ...
latest: Pulling from docker.io/timbordemann/cockpit-kubernetes

I am getting the following error
TASK [Gathering Facts] *********************************************************************************************************
Tuesday 24 March 2020 14:11:44 -0300 (0:00:00.120) 0:00:02.321 *********
fatal: [openshift_cockpit_deployer_image=docker.io/timbordemann/cockpit-kubernetes]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to remote host "openshift_cockpit_deployer_image=docker.io/timbordemann/cockpit-kubernetes". Make sure this host can be reached over ssh", "unreachable": true}
ok: [master.iftikhar.us]

NO MORE HOSTS LEFT *************************************************************************************************************

PLAY RECAP *********************************************************************************************************************
localhost : ok=11 changed=0 unreachable=0 failed=0
master.iftikhar.us : ok=2 changed=0 unreachable=0 failed=0
openshift_cockpit_deployer_image=docker.io/timbordemann/cockpit-kubernetes : ok=0 changed=0 unreachable=1 failed=0

zilmarr commented

@harrisiftikhar I think you're adding that line in the wrong section.
It seems you added to the [hosts] section, you have to add it to the vars section (the end of your inventory file)

harrisiftikhar commented

Thanks! it worked after moving it to the variable section.

harrisiftikhar commented

I am still seeing the following errors.
Failed to pull image "cockpit/kubernetes:latest": rpc error: code = Unknown desc = manifest for docker.io/cockpit/kubernetes:latest not found

I have restarted the ansible deployment after adding openshift_cockpit_deployer_image=docker.io/timbordemann/cockpit-kubernetes:latest
ansible-playbook playbooks/deploy_cluster.yml

harrisiftikhar commented
harrisiftikhar commented

I was able to manually delete the deployment config and recreate it with the timbrd's image (Thanks a lot @timbrd). It'd be nice to figure out if I can understand why the initial deployment was pointing to the wrong image.

"Failed to pull image "cockpit/kubernetes:latest": rpc error: code = Unknown desc = manifest for docker.io/cockpit/kubernetes:latest not found"

Following is my /etc/ansible/hosts file:

[OSEv3:children]
masters
nodes
etcd

[OSEv3:vars]
openshift_deployment_type=origin
os_firewall_use_firewalld=True
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
openshift_pkg_version='-3.9.0'
openshift_master_default_subdomain=origin.iftikhar.us
openshift_disable_check=disk_avalability,memory_availability
#openshift_disable_check=disk_avalability,memory_availability,docker_storage
ansible_ssh_user=root
ansible_user=root
openshift_cockpit_deployer_image='docker.io/timbordemann/cockpit-kubernetes:latest'
#template_service_broker_install=false

.....

tinu73 commented

Solution is described here

khodayard commented

No need to edit ansible host file and redeploy the whole cluster, I've just edited this deployment yaml file and replaced not-working (not-found) image with this:

docker.io/timbordemann/cockpit-kubernetes:latest

It was mentioned in two places.
Register console came up as soon as I've saved the yaml file.
Thanks to @timbrd

openshift-bot commented

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

pgress commented

/remove-lifecycle stale

openshift-bot commented

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot commented

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

mtnbikenc commented

Fixed in #12278