Obsolete CSRs are not deleted when creating new ones
Closed this issue · 1 comments
mook-as commented
Hi!
CF-operator version: 3.1.0+0.gd5ad8a7d
When using a certificate (generated at run time) that has signer_type: cluster, we create a kube-level CSR object. However, if we re-deploy the BDPL, the CSR might already exist; in that case, the existing one blocks the new CSR from being created. However, cf-operator uses the previous signing result, which means we can get mismatched certificate and private key:
$ kubectl get secret kubecf.var-unused-certificate --namespace kubecf -o jsonpath={.data.certificate}
Modulus:
00:c6:45:67:24:42:bc:0c:15:ba:8a:66:ee:e5:d4:
$ kubectl get secret kubecf.var-unused-certificate --namespace kubecf -o jsonpath={.data.private_key}
modulus:
00:c2:77:16:aa:f1:5c:24:ba:03:a4:6e:96:89:47:
repro-ops.yaml
- type: replace
path: /variables/-
value:
name: unused_certificate
options:
ca: service_cf_internal_ca
signer_type: cluster
type: certificate
- # This is just to make sure the cert gets used somewhere.
type: replace
path: /instance_groups/name=nats/jobs/name=nats/properties?/unused
value:
cert: ((unused_certificate.certificate))
key: ((unused_certificate.private_key))Logs
Note that I've reformatted things; this was originally two giant lines, which is unreadable on GitHub. Both have been reformatted (JSON → YAML, for example) for readability.2020-02-28T01:24:14.562Z
DEBUG controller-runtime.manager.events recorder/recorder.go:52 Normal
reason: Predicates
object:
apiVersion: quarks.cloudfoundry.org/v1alpha1
kind: QuarksSecret
name: kubecf.var-unused-certificate
namespace: kubecf
resourceVersion: '86305'
uid: f7b8086b-3275-4ec4-8518-d1a2c3b0f206
message:
message: Create predicate passed for 'kubecf.var-unused-certificate'
namespace: kubecf
predicateObjectKind: qsv1a1.QuarksSecret
predicateObjectName: kubecf.var-unused-certificate
reconciliationObjectKind: qsv1a1.QuarksSecret
reconciliationObjectName: kubecf.var-unused-certificate
type: Predicates
2020-02-28T01:29:49.167Z
ERROR controller-runtime.controller controller/controller.go:258 Reconciler error
controller: quarks-secret-controller
error:
generating certificate secret.:
could not create certificatesigningrequest 'kubecf-kubecfvar-unused-c':
certificatesigningrequests.certificates.k8s.io "kubecf-kubecfvar-unused-c" already exists
errorVerbose:
certificatesigningrequests.certificates.k8s.io "kubecf-kubecfvar-unused-c" already exists
could not create certificatesigningrequest 'kubecf-kubecfvar-unused-c'
code.cloudfoundry.org/cf-operator/pkg/kube/controllers/quarkssecret.(*ReconcileQuarksSecret).createCertificateSigningRequest
/go/src/code.cloudfoundry.org/cf-operator/pkg/kube/controllers/quarkssecret/quarkssecret_reconciler.go:486
code.cloudfoundry.org/cf-operator/pkg/kube/controllers/quarkssecret.(*ReconcileQuarksSecret).createCertificateSecret
/go/src/code.cloudfoundry.org/cf-operator/pkg/kube/controllers/quarkssecret/quarkssecret_reconciler.go:303
code.cloudfoundry.org/cf-operator/pkg/kube/controllers/quarkssecret.(*ReconcileQuarksSecret).Reconcile
/go/src/code.cloudfoundry.org/cf-operator/pkg/kube/controllers/quarkssecret/quarkssecret_reconciler.go:143
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:256
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:88
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357
generating certificate secret.
code.cloudfoundry.org/cf-operator/pkg/kube/controllers/quarkssecret.(*ReconcileQuarksSecret).Reconcile
/go/src/code.cloudfoundry.org/cf-operator/pkg/kube/controllers/quarkssecret/quarkssecret_reconciler.go:150
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:256
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:88
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357
Desired behaviour:
I would like either one of two options:
- The operator deletes the previous CSR and makes a new one.
- The operator detects this situation (possibly by the private key not matching the public key), and rejects it.
cf-gitbot commented
We have created an issue in Pivotal Tracker to manage this:
https://www.pivotaltracker.com/story/show/171530959
The labels on this github issue will be updated when the story is started.