[BUG] GSLB is not updated when Ingress has change

[kuritka]

We found a bug where GSLB does not update itself (specifically annotations section) when the ingress has changed.

The problem with the annotations was that the rancher kept changing the annotations and k8gb kept changing them back. They were stuck in a loop.

The bug is located here:

k8gb/controllers/gslb_controller.go

Lines 234 to 237 in 8820bba

	log.Info().
	Str("ingress", a.GetName()).
	Msg("Ingress does not exist anymore. Skipping Glsb creation...")
	return

• implement an update functionality of the GSLB as the ingress changes.
• Implement the necessary tests to make sure that the GSLB update is done exactly as we need.

[kuritka]

returning back on this task:

BUG reproducing manually

having ONLY simple podinfo app and k8gb deployed (using clean test environment https://github.com/kuritka/k8gb-demo-cncf-2022)

than, by applying ingress puting test application under k8gb control (application is loadbalancable):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    x.y.io/ep: '[{"addresses":["1.2.3.4"],"port":80}]'
    k8gb.io/strategy: roundRobin
  name: gslb
  namespace: demo
spec:
  ingressClassName: nginx
  rules:
  - host: demo.cloud.example.com
    http:
      paths:
      - backend:
          service:
            name: frontend-podinfo
            port:
              name: http
        path: /
        pathType: Prefix

manually changing ingress k -n demo edit ing gslb and change annotation to x.y.io/ep: '[{"addresses":["2.2.2.2"],"port":8080}]'

command k -n demo get ing -o=jsonpath='{.items[*].metadata.annotations}{"\n"}' returns unchanged annotation
map[x.y.io/ep:[{"addresses":["1.2.3.4"],"port":80}]

[kuritka]

related to #1019

This error arises because GLSB and ingress are two sources for k8gb. If you overwrite one of them, k8gb takes the data from the second one and overwrites the first one. Another consequence is that if you delete one source, for example, it can be regenerated from the first one. If there is an inconsistency, one, the source is modified by the other and vice versa. This can cause several error cycles at the beginning until the reconciliation is stopped. It also depends on whether you create an ingress with GSLB or vice versa.

The best solution is to have only one source of truth (in our case it would be ingress, later gateway).

I started a working demo where I removed this : k8gb-lite. This required refactoring the watcher, throwing out the GSLB and rewriting the terratest. I put terratests for FO, RR, WRR against three clusters and did manual tests at the same time, everything works as expected.