canonical/seldon-core-operator

seldon-controller-manager failed to upgrade 1.6 to 1.7

i-chvets opened this issue · 1 comments

i-chvets commented

Description

seldon-controller-manager failed to upgrade 1.6 to 1.7
Failed to reach active/idle:

seldon-controller-manager blocked: K8S resources creation failed

Jira

Merge into:

  • track/1.15 (release KF v1.7)
  • main

Testing

Deploy charm from stable:

juju deploy seldon-core seldon-controller-manager --channel=1.14/stable --trust

Verify that deployment:

juju status
Model         Controller  Cloud/Region        Version  SLA          Timestamp
test-upgrade  uk8s        microk8s/localhost  2.9.34   unsupported  11:25:23-04:00

App                        Version                Status  Scale  Charm        Channel      Rev  Address        Exposed  Message
seldon-controller-manager  res:oci-image@eb811b6  active      1  seldon-core  1.14/stable   92  10.152.183.81  no       

Unit                          Workload  Agent  Address      Ports              Message
seldon-controller-manager/0*  active    idle   10.1.59.111  8080/TCP,4443/TCP

Build local charm and execute refresh command:

juju refresh seldon-controller-manager --path=./seldon-core_ubuntu-20.04-amd64.charm --resource="oci-image=docker.io/seldonio/seldon-core-operator:1.15.0"

Verify that upgrade was successful:

juju status
Model         Controller  Cloud/Region        Version  SLA          Timestamp
test-upgrade  uk8s        microk8s/localhost  2.9.34   unsupported  11:27:41-04:00

App                        Version                         Status  Scale  Charm        Channel  Rev  Address         Exposed  Message
seldon-controller-manager  .../c5e3s519ko1quc9tqnysy92...  active      1  seldon-core  stable     0  10.152.183.252  no       

Unit                          Workload  Agent  Address      Ports  Message
seldon-controller-manager/0*  active    idle   10.1.59.112
DnPlas commented

I was able to get the same message:
seldon-controller-manager/0* blocked idle 10.1.216.35 K8S resources creation failed

Steps to reproduce

  1. juju deploy seldon-core seldon-controller-manager --channel 1.6/stable --trust #wait until it's active and idle
  2. juju refresh seldon-controller-manager --channel latest/edge --trust
  3. training-operator should be in BlockedStatus

Let's dig a bit deeper in the charm code to figure out this one.