Unexpected behavior when changing the plugin of a `ClusterScan`

[matheusfm]

Description

It's a very specific bug that happens when we change the plugin of a ClusterScan.

How to Reproduce

1. Create a Cluster and a ClusterScan (popeye plugin by default):

cat << EOF | kubectl apply -f -
apiVersion: zora.undistro.io/v1alpha1
kind: ClusterScan
metadata:
  name: foo
spec:
  clusterRef:
    name: foo
  schedule: "* * * * *"
EOF

2. Wait for a successful scan (19 issues):

NAME   CLUSTER   SCHEDULE    SUSPEND   PLUGINS   LAST STATUS   LAST SCHEDULE   LAST SUCCESSFUL   ISSUES   READY   SAAS   AGE
foo    foo       * * * * *   false     popeye    Complete      53s             44s               19       True    OK     26m

3. Create a plugin that always fails:

cat << EOF | kubectl apply -f -
apiVersion: zora.undistro.io/v1alpha1
kind: Plugin
metadata:
  name: brutus
  namespace: zora-system
spec:
  command:
    - /bin/sh
    - -c
    - |
      echo "ERROR" > \$(DONE_DIR)/error
      exit 27
  image: docker.io/derailed/popeye:v0.11.1
EOF

4. Update the ClusterScan to use this plugin:

cat << EOF | kubectl apply -f -
apiVersion: zora.undistro.io/v1alpha1
kind: ClusterScan
metadata:
  name: foo
spec:
  plugins:
    - name: brutus
  clusterRef:
    name: foo
  schedule: "* * * * *"
EOF

5. The change does not reflect on ClusterScan that keep using popeye

NAME                               CLUSTER   SCHEDULE    SUSPEND   PLUGINS   LAST STATUS   LAST SCHEDULE   LAST SUCCESSFUL   ISSUES   READY   SAAS   AGE   NEXT SCHEDULE
clusterscan.zora.undistro.io/foo   foo       * * * * *   false     popeye    Complete      7m56s           7m47s             19       True    OK     39m   2023-03-07T13:56:00Z

NAME                       SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE   CONTAINERS      IMAGES                                                                  SELECTOR
cronjob.batch/foo-popeye   * * * * *   False     0        56s             39m   worker,popeye   ghcr.io/undistro/zora/worker:v0.4.4,docker.io/derailed/popeye:v0.11.1   <none>

NAME                            READY   STATUS      RESTARTS   AGE     IP            NODE                 NOMINATED NODE   READINESS GATES
pod/foo-popeye-27969960-t86sn   0/2     Completed   0          2m56s   10.244.0.39   kind-control-plane   <none>           <none>
pod/foo-popeye-27969961-d6qx4   0/2     Completed   0          116s    10.244.0.40   kind-control-plane   <none>           <none>
pod/foo-popeye-27969962-lfctl   0/2     Completed   0          56s     10.244.0.41   kind-control-plane   <none>           <none>

Expected Behavior

The expected behavior is that the ClusterScan uses the new plugin.

System Information

• Kubernetes v1.25.3
• Zora 0.4.4