Game Plan for kube-external-sync

[alehechka]

The inspiration for this application comes from the design of @dmsi-io's Kubernetes cluster use feature branches. Whenever a developer would push commits to a new branch (admittedly we should have changed this to PRs only), it will run a job that uses gha-k8s-namespace which will handle the spin-up of certain resources to make that new feature namespace all-encompassing. The premise was that for every Service defined in the default (for us it was develop) namespace, it will create an ExternalName service in the new namespace that simply points to the original Service in the default namespace. This will create a DNS entry in the new namespace that any application will be able to call in the same way to be able to talk to the application that is only deployed in that default namespace.

Example:

• Request to http://example-api in example namespace will be redirected to example-api.default.svc.cluster.local which is the original Service for example-api in the default namespace

Additionally, that gha-k8s-namespace action copies/manipulates all ingresses and creates them in the new namespace, as well as copies many other resources, and fully deploys certain dependency apps such as gateway applications.

The goal of this application is to automate the creation of ExternalName Services and Ingresses to the new feature branch namespaces. This will take a similar approach to kube-secret-sync in which there will be a CRD that is used to define which Service/Ingress to sync, then the application will watch all CRDs, Services, and Ingresses, and under the right conditions will create the ExternalName and Ingress with slight adjustments for the new namespace.

The application will also make an assumption with namespace names and domains. One of the variables provided to the CRD spec will be the wildcard base domain, and when syncing resources to a new namespace, the namespace name will be substituted in for the first subdomain of the top-level domain.

Example:

• Provided TLD: *.example.com
• New namespace: new-feature
• New TLD: new-feature.example.com

By default, this application will only support core Service and Ingress resources from the Kubernetes API. However, I already have plans to turn-on optional support for Traefik CRDs for Services and Ingresses.

The spec for the CRD for this application will look as follows:

spec:
  namespace: 'namespace-of-resources'
  service: 
    name: 'name-of-resource'
    kind: 'Service'
    externalNameSuffix: 'svc.cluster.local' 
  ingress:
    name: 'name-of-ingress'
    kind: 'Ingress'
    topLevelDomain: '*.example.com'
  rules:
    namespaces:
      exclude: 
        - kube-system
        - kube-public
      excludeRegex: 
        - 'kube[.]*'
      include: 
        - default
      includeRegex: 
        - 'sync-[.]*'
      includeAnnotation:
        - key: 'some-annot-key'
          value: '*'
          regex: true

Unlike the CRD for kube-secret-sync, the rules for this CRD do not include an option for force. The reason for this is to be non-intrusive. So if a Service or Ingress of the same name/namespace already exists, this application will only ever attempt to update/delete if the resources are already being managed by this application.

[alehechka]

I just found this sweet Kubernetes app (that, unfortunately, does everything that kube-secret-sync does plus more), but it has a really cool pattern where it uses exclusively annotations to determine what and where to sync resources.

I think it would be a waste to redo kube-secret-sync, since that app already does everything (likely just archive the repo tbh), but I think it would be worthwhile to convert this application to follow the same annotation-centered pattern. The original thought around the CRDs was to pull the configuration of the app to a single scoped item, and I was under the assumption that CRDs were the best way to do so. But, from further experience, it seems that annotations are a very regular and standard way of doing things and allow the configuration to be so close to the actual resource that they end up within the resource!

I just finished up the sync of Services and Ingresses with the use of ExternalSyncRules, and had planned on adding the Traefik sync next. But now, I think it would be better to "scrap" the current code as a v0, start with a new unreleased version going forward, and eventually release that as a v1. This will be a rough transition at first and I will likely delete A LOT of code, but I think it will make for a better product. On top of the annotation-centered approach, I am also going to replicate their controller model which encapsulates each resource type into its own controller and has a generic one that implements it and orchestrates synchronization.