Tf-runner pod not created, kind Terraform stuck in "Reconciliation in progress"

[manicole]

Hi all,
I'm trying to deploy a kind Terraform via a kind Kustomization, but often no tf-runner is created and my Terraform stays stuck in "Reconciliation in progress" state.

I'm trying to deploy this Kustomization:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: k3s
  namespace: k3s
spec:
  targetNamespace: k3s
  interval: 10m
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: gitrepo
    namespace: flux-system
  path: /k3s/manifests
  prune: true
  decryption:
    provider: sops
    secretRef:
      name: sops-age
  postBuild:
    substitute:
      service: k3s
      previous: openstack
      target: k3s
      destroy: "false"

In gitrepo/k3s/manifests, there is only my kind Terraform:

apiVersion: infra.contrib.fluxcd.io/v1alpha2
kind: Terraform
metadata:
  name: ${service}
spec:
  alwaysCleanupRunnerPod: false
  serviceAccountName: ${target}
  interval: 1m
  destroy: ${destroy}
  destroyResourcesOnDeletion: true
  approvePlan: auto
  path: "/k3s/terraform"
  sourceRef:
    kind: GitRepository
    name: gitrepo
    namespace: flux-system
   dependsOn:
   - name: ${previous}
  vars:
  - name: service
    value: ${service}
  - name: previous
    value: ${previous}
  - name: target
    value: ${target}
  varsFrom:
  - kind: Secret
    name: ${previous}-output
    varsKeys:
    - instance_ip
    - instance_ssh_key
  writeOutputsToSecret:
    name: ${service}-output
  runnerPodTemplate:
    spec:
      volumes:
      - name: tmp
        emptyDir: {}
      volumeMounts:
      - name: tmp
        mountPath: "/tmp"

What happens:

1. I apply the Kustomization file
2. Kustomization is deployed and ready
3. Terraform is deployed, state "Unknown" and status "Reconciliation in progress"
4. Nothing appears in Namespace k3s, although I would expect a k3s-tf-runner to be deployed...

Additionnal info:

• I deploy plenty of Kustomization files, and do not have any problem with the others.

• Sometimes by chance (non reproducible pattern) a k3s-tf-runner is created and everything works fine till the end ; a few hours later, in spite of the arg alwaysCleanupRunnerPod: false, the k3s-tf-runner disappears ... and I can't create it back.

• I can't find any clue of what's happening in tf-controller logs nor in namespace events.

• All my kind Terraform does (gitrepo/k3s/terraform/main.tf) is a local exec:

resource "null_resource" "k3s" {
  provisioner "local-exec" {
    command = <<EOT
    mkdir -p ${var.instance_ssh_folder}
    echo -e '${var.instance_ssh_key}' >${var.instance_ssh_folder}/${var.instance_ssh_file}
    mkdir -p ${var.instance_kubeconfig_folder}
    touch ${var.instance_kubeconfig_folder}/${var.instance_kubeconfig_file}
    wget https://github.com/alexellis/k3sup/releases/download/0.13.6/k3sup
    chmod +x k3sup
    ./k3sup install \
    --ip ${var.instance_ip} \
    --user ${var.instance_user} \
    --ssh-key ${var.instance_ssh_folder}/${var.instance_ssh_file} \
    --cluster \
    --local-path ${var.instance_kubeconfig_folder}/${var.instance_kubeconfig_file} \
    --context ${var.instance_kubeconfig_context}
    EOT
  }
}

• Versions:

$ kubectl version
Client Version: v1.29.3+k3s1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.3+k3s1

$ helm ls -n flux-system
NAME         	NAMESPACE  	REVISION	UPDATED                              	STATUS  	CHART                       	APP VERSION
tf-controller	flux-system	1       	2024-06-19 13:13:30.0861225 +0000 UTC	deployed	tf-controller-0.16.0-alpha.3	v0.16.0-alpha.3

Thanks in advance for helping :)

[manicole]

I found a new element.

Commenting the runnerPodTemplate spec from my kind Terraform makes k3s-tf-runner appear (once FluxCD reconciles my Kustomization).
Uncommenting it afterwards does not seem to be a problem, but who knows how long for.

[manicole]

Anyone ?

I still have the issue: I have to apply my kind Terraform without any runnerPodTemplateto launch the tf-runner Pod (otherwise, it never appears). Once launched, I can add a runnerPodTemplate to my my kind Terraform and reapply to update it.

[akselleirv]

Hello @manicole ,
Any specific reason you need to define the tmp volume in the runner pod template?

[manicole]

Hi @akselleirv, thanks for reacting.

Actually I am still trying to solve a problem, and thought it was the way. I install k3s with k3sup (i.e. using ssh) and ouput the kubeconfig from the Terraform plan:

# outputs.tf
data "local_file" "kubeconfig_file" {
  filename   = "${var.instance_kubeconfig_folder}/${var.instance_kubeconfig_file}"
  depends_on = [null_resource.k3s]
}

output "kubeconfig_file" {
  description = "kubeconfig to access k3s cluster"
  value       = nonsensitive(data.local_file.kubeconfig_file.content)
}

I get the following error, and thought mounting tmp would be enough but it is not:

Error: Read local file data source error

  with data.local_file.kubeconfig_file,
  on outputs.tf line 1, in data "local_file" "kubeconfig_file": 
   1: data "local_file" "kubeconfig_file" { 

The file at given path cannot be read.

  Original Error: open /tmp/.kube/config: no such file or directory

This might be another issue to solve for me, but I believe it has no consequences on the problem here.
Thanks