emqx/emqx-operator

Upgrade path from operator 2.1.4 to 2.2.1 unclear

clive-jevons opened this issue · 2 comments

clive-jevons commented

Describe the bug
It is unclear to me on how to upgrade from an existing setup using operator 2.1.4 / broker 5.0.23 to a setup using operator 2.2.1 / broker 5.1.3.

When I upgrade the operator first, it starts creating the new core and then replicant nodes, but never deletes the old core STS / replicant deployment.

When I upgrade the broker first, it never finishes upgrading because of the bug in operator 2.1.4.

🤔

See also #897 (specifically last comment, which I more or less copies as the description here; was not able to re-open the issue, so created a new one).

Could you please outline what the recommended procedure is? Thank you very much in advance 🙏🏻

To Reproduce
Setup a deployment using operator 2.1.4 / broker 5.0.23, then try to upgrade to using operator 2.2.1 / broker 5.1.3

Expected behavior
There is a clear upgrade path which results in a clean state using the new versions.

Anything else we need to know?:

Environment details::

  • Kubernetes version: 1.26
  • Cloud-provider/provisioner: AKS
  • emqx-operator version: 2.1.4 / 2.2.1
  • Install method: helm
Rory-Z commented

Hi, @clive-jevons Sorry for delay.
I'm sorry, but there are some break changes from EMQX 5.0 to EMQX 5.1, so we need to do some manual operations.

Please note that the upgrade process will cause MQTT service interruption.

  1. Deploy EMQX Operator 2.1.4, and deploy EMQX CR like this:
apiVersion: apps.emqx.io/v2alpha1
kind: EMQX
metadata:
  name: emqx
spec:
  image: emqx:5.0.23
  coreTemplate:
    spec:
      replicas: 2
  replicantTemplate:
    spec:
     replicas: 3
  1. Change EMQX image, and Upgrade EMQX Operator by helm
$ kubectl patch emqx emqx --type=merge -p '{"spec": {"image": "emqx:5.1.3"}}'

$ helm upgrade emqx-operator emqx/emqx-operator --namespace emqx-operator-system --create-namespace --version 2.2.1
  1. Wait EMQX status to Available
$ kubectl get emqx.v2beta1.apps.emqx.io emqx
NAME   STATUS   AGE
emqx   Available    8m59s
  1. Delete old sts and deployment
$ kubectl delete sts emqx-core
statefulset.apps "emqx-core" deleted
$ kubectl delete deployments.apps emqx-replicant
deployment.apps "emqx-replicant" deleted
  1. Wait EMQX status is Ready
$ kubectl get emqx.v2beta1.apps.emqx.io emqx
NAME   STATUS   AGE
emqx   Ready    8m59s
clive-jevons commented

Hi @Rory-Z ,

Thank you very much for the info 😁
I've verified the process in a local minikube setup. Will maybe have a go at adding a helm post-install hook in our setup to automate those manual steps 😬

  • 🎉1