GitHub App from secret no longer working

[kasey-weirich]

Curious if anyone else is seeing this behavior:

Over the weekend our runner pool was not available, did some initial digging in logs and found the runner operator throwing the following error:

jsonPayload: {
msg: "Reconciler error"
GithubActionRunner: {
namespace: "github-runners"
name: "runner-pool-primary"
}
controllerGroup: "garo.tietoevry.com"
stacktrace: "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:226"
reconcileID: "3747e140-90c9-498c-98d5-542ef7cacce1"
level: "error"
name: "runner-pool-primary"
controller: "githubactionrunner"
controllerKind: "GithubActionRunner"
error: "could not parse private key: invalid key: Key must be a PEM encoded PKCS1 or PKCS8 key"
namespace: "github-runners"
ts: "2023-10-30T21:27:33Z"
}

Tried re-creating the secret several times but continued to get this error each time I used the secret approach (following the docs here).

Had to eventually hard-code the private key and integration ID directly in the values file to get the operator to work again.

I did this routine back-and-forth several times (use existingSecret, hard-code the privateKey and integrationId). After toggling these configs/settings and re-deploying the operator with each, only the hard-coded values seems to work. I even went as far as copying the secret that was created by the Helm chart and still saw the pem encoding error. Only hard-coding seems to work now.

Curious if other folks have seen this behavior. I saw in another issue that this has come up before.

All configs/resources are in the same namespace.