[Bug]: Health checks fail on flux_bootstrap_git when patching GitRepository name

Question

[Bug]: Health checks fail on flux_bootstrap_git when patching GitRepository name

patrikkj opened this issue 4 months ago · 4 comments

Describe the bug

Health checks fail when patching the GitRepository name. Resources are reporting as healthy in the cluster but health checks assume that the repository has the same name as the flux bootstrapping namespace. When overriding the namespace (namespace = "flux") the error persists but looking for a repository named flux in this case.

Steps to reproduce

Use kustomize to patch the GitRepository name and reference.
Observe that resources are deployed in the cluster but terraform errors with a failing health check.

Expected behavior

Health checks recognize the patched resource names and run against the correct resource names.

Screenshots and recordings

Error when applying resources

Kustomization is created successfully

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  creationTimestamp: "2024-08-25T17:44:27Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 1
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-system
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: flux-system
  namespace: flux-system
  resourceVersion: "1334"
  uid: a32987dc-5a4f-4ccf-9da6-19c1407dc70e
spec:
  force: false
  interval: 10m0s
  path: ./cluster
  prune: true
  sourceRef:
    kind: GitRepository
    name: mono
status:
  conditions:
  - lastTransitionTime: "2024-08-25T17:44:59Z"
    message: 'Applied revision: main@sha1:69f215a963e3bd7da3b757607ee2f378215f657a'
    observedGeneration: 1
    reason: ReconciliationSucceeded
    status: "True"
    type: Ready

GitRepository is also created successfully

Name:         mono
Namespace:    flux-system
Labels:       kustomize.toolkit.fluxcd.io/name=flux-system
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  <none>
API Version:  source.toolkit.fluxcd.io/v1
Kind:         GitRepository
Metadata:
  Creation Timestamp:  2024-08-25T17:44:27Z
  Finalizers:
    finalizers.fluxcd.io
  Generation:        1
  Resource Version:  1273
  UID:               766fa93f-ff7c-4bb3-81e1-e631ddb27f7d
Spec:
  Interval:  1m0s
  Ref:
    Branch:  main
  Secret Ref:
    Name:   flux-system
  Timeout:  60s
  URL:      ssh://git@github.com/<orgname>/mono.git
Status:
  Artifact:
    Digest:            sha256:4d057b06377622801f2a4a7a51e7b0b076ff0aea11187a5380853ab5db946501
    Last Update Time:  2024-08-25T17:44:31Z
    Path:              gitrepository/flux-system/mono/69f215a963e3bd7da3b757607ee2f378215f657a.tar.gz
    Revision:          main@sha1:69f215a963e3bd7da3b757607ee2f378215f657a
    Size:              202463
    URL:               http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/mono/69f215a963e3bd7da3b757607ee2f378215f657a.tar.gz
  Conditions:
    Last Transition Time:  2024-08-25T17:44:31Z
    Message:               stored artifact for revision 'main@sha1:69f215a963e3bd7da3b757607ee2f378215f657a'
    Observed Generation:   1
    Reason:                Succeeded
    Status:                True
    Type:                  Ready
    Last Transition Time:  2024-08-25T17:44:31Z
    Message:               stored artifact for revision 'main@sha1:69f215a963e3bd7da3b757607ee2f378215f657a'
    Observed Generation:   1
    Reason:                Succeeded
    Status:                True
    Type:                  ArtifactInStorage
  Observed Generation:     1

Terraform and provider versions

Terraform v1.5.5
on darwin_arm64

Terraform provider configurations

provider "flux" {
  kubernetes = {
    ...
  }
  git = {
    url = local.vars.repo.url
    ssh = {
      username    = "git"
      private_key = tls_private_key.repo.private_key_pem
    }
  }
}

flux_bootstrap_git resource

resource "flux_bootstrap_git" "this" {
  embedded_manifests     = true
  path                   = "./cluster"
  kustomization_override = <<-EOF
    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    resources:
      - gotk-components.yaml
      - gotk-sync.yaml
    patches:
      - patch: |
          - op: replace
            path: /metadata/name
            value: mono
        target:
          kind: GitRepository
          name: flux-system
      - patch: |
          - op: replace
            path: /spec/sourceRef/name
            value: mono
        target:
          kind: Kustomization
          name: flux-system
  EOF

  depends_on = [helm_release.cilium]
}

Flux version

v2.3.0

Additional context

Output from `terraform apply`

flux_bootstrap_git.this: Creating...
flux_bootstrap_git.this: Still creating... [10s elapsed]
flux_bootstrap_git.this: Still creating... [20s elapsed]
flux_bootstrap_git.this: Still creating... [30s elapsed]
╷
│ Error: Bootstrap run error
│ 
│   with flux_bootstrap_git.this,
│   on 02-flux.tf line 33, in resource "flux_bootstrap_git" "this":
│   33: resource "flux_bootstrap_git" "this" {
│ 
│ bootstrap failed with 1 health check failure(s): error while waiting for GitRepository to be ready: 'gitrepositories.source.toolkit.fluxcd.io "flux-system" not found'
╵

Versions

$ flux version                                     
flux: v2.3.0
distribution: flux-v2.3.0
helm-controller: v1.0.1
kustomize-controller: v1.3.0
notification-controller: v1.3.0
source-controller: v1.3.0

Code of Conduct

I agree to follow this project's Code of Conduct

Would you like to implement a fix?

None

Answer 1 · 2024-08-25T18:39:27.000Z

It would be a great enhancement if the repository name could be passed as an input to the flux_bootstrap_git resource. This would allow users to adapt the GitRepository name to the actual name of the repository, and prevent issues like the health check failures mentioned above.

Answer 2 · 2024-09-12T11:07:48.000Z

@patrikkj, apologies for the long delay.

Once a cluster is bootstrapped using the provider, it's recommended that you avoid changing the bootstrap kustomization in Terraform.

For better management and consistency, it's highly recommended that all GitRepository resources be included in the same Git repository the provider uses for bootstrapping. These resources should be managed directly in your repository to ensure proper version control and synchronisation.

The bootstrap provider is intended to initialise the cluster once, and the flux_bootstrap_git resource definition should ideally remain unchanged afterwards.

You can find an example of where the GitRepository resources are stored in this repository. Additionally, the primary GitRepository for the flux-system namespace during bootstrapping can be seen here, which remains untouched after the initial bootstrap.

Answer 3 · 2024-09-15T19:28:34.000Z

I am very much in favor of this improvement! 👍🏼

@swade1987 Managing multiple clusters with a "bootstrap" repository is great, but I guess this does not take another use case into account: Flux setups that are only targeted for a single cluster or designed to be a all-in-one repository. This is the monorepo layout that i described in the official Flux documentation. I use this setup to maintain multiple clusters by simply having multiple "cluster-sync" Kustomizations per cluster that recursively patches other Kustomizations that should go into the desired cluster.
Having the GitRepository and Kustomization "hard-coded" named to flux-system does not match the name of the monorepo, making it inconsistent with the rest of the setup and is also confusing for developer teams that are not deeply involved into GitOps.

Allowing to set the name or disable the health checks (or ignore their outcome) would definitely help towards the goal of supporting all officially documented repository layouts.