Not able to obtain TLS certificate using step-issuer
sanjeevrg89 opened this issue · 13 comments
Hi Mariano, As discussed on Twitter I am submitting an issue here.
I have followed the documentation and implemented steps as suggested:
My environment details:
1. Running AKS kubernetes Cluster
2. cert-manager is deployed on the AKS cluster (version is v0.10.0)
3. Installed step-certificates helm chart
{ helm repo add smallstep https://smallstep.github.io/helm-charts
helm repo update
helm install --name step-certificates smallstep/step-certificates }
-
Installed Step-Issuer
kubectl apply -f config/crd/bases
kubectl apply -f config/samples/deployment.yaml -
Created a StepIssuer object with default provisioner(admin), kid
6. Created certificate object
certificate.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: xxx
namespace: default
spec:
secretName: xxx-tls
commonName: 'guest.xxx.net'
dnsNames:
- 'guest.xxx.net'
duration: 72h
renewBefore: 8h
issuerRef:
group: certmanager.step.sm
kind: CertificateRequest
name: step-issuer
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
cert-manager-585c7c5bc7-xfx9x 1/1 Running 10 3d21h
cert-manager-cainjector-6f868ccdf6-hz6lp 1/1 Running 95 3d21h
cert-manager-webhook-5896b5fb5c-mgx5c 1/1 Running 2 3d21h
step-certificates-0 1/1 Running 0 3d21h
step-certificates-z95m7 0/1 Completed 0 3d21h
$ kubectl get stepissuer
NAME AGE
step-issuer 3d23h
$ kubectl describe stepissuer step-issuer
Status:
Conditions:
Last Transition Time: 2019-09-19T20:23:05Z
Message: StepIssuer verified and ready to sign certificates
Reason: Verified
Status: True
Type: Ready
Events:
$ kubectl get certificate
NAME READY SECRET AGE
xxx xxx-tls 3d21h
$ kubectl describe certificate xxx
Name: xxx
Namespace: default
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"Certificate","metadata":{"annotations":{},"name":"xxx","namespace":"defau...
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata:
Creation Timestamp: 2019-09-19T20:27:17Z
Generation: 1
Resource Version: 859688
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/xxx
UID: dfedd671-db1b-11e9-ae22-9a4e0da330d0
Spec:
Common Name: guest.xxx.net
Dns Names:
guest.xxx.net
Duration: 72h0m0s
Issuer Ref:
Group: certmanager.step.sm
Kind: CertificateRequest
Name: step-issuer
Renew Before: 8h0m0s
Secret Name: xxx-tls
Status:
Events:
I dont see a certificate being generated and stored in kubernetes secret.
Request you to look into this and help us out. @maraino
Hi @devops2201: Have you enabled --feature-gates=CertificateRequestControllers=true in the cert-manager pod. This is not yet enabled by default on cert-manager 0.10.0.
And another question, do you have problems using CertificateRequest resources to get a TLS certificate?
To set the flag you need to edit the cert-manager.yaml and add in the cert-manager deployment - --feature-gates=CertificateRequestControllers=true in the args property. It should look like:
# Source: cert-manager/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: cert-manager
namespace: "cert-manager"
labels:
app: cert-manager
app.kubernetes.io/name: cert-manager
app.kubernetes.io/instance: cert-manager
app.kubernetes.io/managed-by: Tiller
helm.sh/chart: cert-manager-v0.10.0
spec:
replicas: 1
selector:
matchLabels:
app: cert-manager
app.kubernetes.io/name: cert-manager
app.kubernetes.io/instance: cert-manager
app.kubernetes.io/managed-by: Tiller
template:
metadata:
labels:
app: cert-manager
app.kubernetes.io/name: cert-manager
app.kubernetes.io/instance: cert-manager
app.kubernetes.io/managed-by: Tiller
helm.sh/chart: cert-manager-v0.10.0
annotations:
prometheus.io/path: "/metrics"
prometheus.io/scrape: 'true'
prometheus.io/port: '9402'
spec:
serviceAccountName: cert-manager
containers:
- name: cert-manager
image: "quay.io/jetstack/cert-manager-controller:v0.10.0"
imagePullPolicy: IfNotPresent
args:
- --v=2
- --cluster-resource-namespace=$(POD_NAMESPACE)
- --leader-election-namespace=$(POD_NAMESPACE)
- --webhook-namespace=$(POD_NAMESPACE)
- --webhook-ca-secret=cert-manager-webhook-ca
- --webhook-serving-secret=cert-manager-webhook-tls
- --webhook-dns-names=cert-manager-webhook,cert-manager-webhook.cert-manager,cert-manager-webhook.cert-manager.svc
- --feature-gates=CertificateRequestControllers=true
ports:
- containerPort: 9402
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
resources:
requests:
cpu: 10m
memory: 32Mi@devops2201 applying that YAML will probably force a redeployment of the pod with the feature enabled.
@maraino I just enabled the argument(--feature-gates=CertificateRequestControllers=true) in the cert manager deployment and cert-manager pod got recreated with the argument.
I reapplied the certificate object which is kind of working now and I see the status of the certificate as FALSE
$ kubectl get certificate
NAME READY SECRET AGE
xxxdev False xxxdev-tls 95s
cert-manager pod logs:
0 1 certificate_request.go:606] cert-manager/controller/certificates/certificates-experimental/certificates-experimental "level"=0 "msg"="CertificateRequest is not in a final state, waiting until CertificateRequest is complete" "related_resource_kind"="CertificateRequest" "related_resource_name"="xxx-4153323434" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="xxx" "resource_namespace"="default" "state"=""
After certain retries: certificaterequest is deleted from the cluster
Log:
E0923 19:09:02.837326 1 controller.go:189] cert-manager/controller/certificaterequests-issuer-acme "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.certmanager.k8s.io "xxx-4153323434" not found"
To answer your question: And another question, do you have problems using CertificateRequest resources to get a TLS certificate?
-> I had tried this earlier by not enabling the argument that you suggested but I can give a retry now that I have enabled the argument.
With CertificateRequest, I will have to create a CSR and base64 encode it and then pass it in the YAML definition.
With Certificate object it helps us as it creates the CSR for us and we dont have to go through that process
With this YAML:
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: test-smallstep-com
namespace: default
spec:
# The secret name to store the signed certificate
secretName: test-smallstep-com-tls
# Common Name
commonName: test.smallstep.com
# DNS SAN
dnsNames:
- localhost
- test.smallstep.com
# IP Address SAN
ipAddresses:
- "127.0.0.1"
# Duration of the certificate
duration: 24h
# Renew 8 hours before the certificate expiration
renewBefore: 8h
# The reference to the step issuer
issuerRef:
group: certmanager.step.sm
kind: CertificateRequest
name: step-issuerYou should see a CertificateRequest, a Certificate and a Secret being created:
$ kubectl apply -f /tmp/test.yaml
certificate.certmanager.k8s.io/test-smallstep-com created
$ kubectl get certificaterequest
NAME READY AGE
test-smallstep-com-920567257 True 6s
$ kubectl get certificate
NAME READY SECRET AGE
test-smallstep-com True test-smallstep-com-tls 14s
$ kubectl get secrets test-smallstep-com-tls -o yaml
apiVersion: v1
data:
ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJqVENDQVRLZ0F3SUJBZ0lSQUtWN0xSZDVQLzY3VVZGdy9pVDNaajB3Q2dZSUtvWkl6ajBFQXdJd0pERWkKTUNBR0ExVUVBeE1aVTNSbGNDQkRaWEowYVdacFkyRjBaWE1nVW05dmRDQkRRVEFlRncweE9UQTVNak14T0RJeQpNemhhRncweU9UQTVNakF4T0RJeU16aGFNQ1F4SWpBZ0JnTlZCQU1UR1ZOMFpYQWdRMlZ5ZEdsbWFXTmhkR1Z6CklGSnZiM1FnUTBFd1dUQVRCZ2NxaGtqT1BRSUJCZ2dxaGtqT1BRTUJCd05DQUFTV1Y1eWdNL0tNVmVUU0pUc1QKQUdsMU4yR0dDaSs4eFZVMHJ1NEZTd1ZxdllVcW0yRXBtZkRIek9xK0laQlR1dWNRam1LVU5QdEFmZ2x5MHhwZApHTWYrbzBVd1F6QU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFUQWRCZ05WCkhRNEVGZ1FVUkpNSUo4c3VhWWZQb09nZDFMQno2UVR3RnRzd0NnWUlLb1pJemowRUF3SURTUUF3UmdJaEFQTXYKN2FVWnlJdkhpUXEya0xBWEdVYXZ1cjRjdGx0VGgrOGZHd1hhWnNZRUFpRUE5aEFYdzFWNnM1M2MvUHF3Qzc0VAowSHhLMk9PSTJUbHIvNjFqYldSLzlNND0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHVENDQXI2Z0F3SUJBZ0lSQU5kQXIrUjc4QVAzY29GdmpadzN5KzB3Q2dZSUtvWkl6ajBFQXdJd0xERXEKTUNnR0ExVUVBeE1oVTNSbGNDQkRaWEowYVdacFkyRjBaWE1nU1c1MFpYSnRaV1JwWVhSbElFTkJNQjRYRFRFNQpNRGt5TXpFNU1UUXlORm9YRFRFNU1Ea3lOREU1TVRReU5Gb3dOREVWTUJNR0ExVUVDaE1NWTJWeWRDMXRZVzVoCloyVnlNUnN3R1FZRFZRUURFeEowWlhOMExuTnRZV3hzYzNSbGNDNWpiMjB3Z2dFaU1BMEdDU3FHU0liM0RRRUIKQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURQS0FNcDl4MnpIcGZXOFZlbTRVb0NmU2d5b3l6TzdKOXk1Nm5DNml5OAprWVlvS1hjclhhdmtYWXVsTFVoc1gyVUFiTVVJSnlHWTZuM3dXaTF2ZHpubE1TWlE0Z3BaK1VtcGo2QUowbjQvCmlRak15SjlrN2RkZnZiMWJqNFdEc1NhQmdtUjlUSlNqTXNIZEhrb2lpSmErZEQybmJvampIZitwOUhGeXo0ZWkKeWZnZzBGejNCL3FMajAwbU5SOFljSlVSN09VNFBUU2V4RG1uTCtNQTZJVlZ2OHl0ZG05d3BrL3drYXhGSExRcQpPK3NZdkJwanR4bVBTb1UwUmhtWFZnOHN2dzZzZUdUUWt2cTZLK3MvWkFyamtlRnJvYkUvSFdZdjdQWndqdzZrCkVVenY2N3ZYT0MySkFhMWpTWk1xbU1OZFVSQWVqcGx6SmpySXZEb0tiaGVkQWdNQkFBR2pnZTB3Z2Vvd0RnWUQKVlIwUEFRSC9CQVFEQWdXZ01CMEdBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFkQmdOVgpIUTRFRmdRVWZRZ2h2N2d4NHZhM2JmM2hSdXV3MjB6Y0ltUXdId1lEVlIwakJCZ3dGb0FVaDlqQXVVZlV0RzRzClQ3VUhBWFIwcGZmcVlwVXdMZ1lEVlIwUkJDY3dKWUlTZEdWemRDNXpiV0ZzYkhOMFpYQXVZMjl0Z2dsc2IyTmgKYkdodmMzU0hCSDhBQUFFd1NRWU1Ld1lCQkFHQ3BHVEdLRUFCQkRrd053SUJBUVFGWVdSdGFXNEVLMWxaTUdodApiRE5PTlZoSVNuaDBWR1ZqYm5WalpYZFlOVmxTY1hvME1pMVRhVkpoVTJSTVprc3hNMnN3Q2dZSUtvWkl6ajBFCkF3SURTUUF3UmdJaEFML1ZNcURueVU3L3VJYllmRjhWYWV2eVZlaEhGejFIcithRmtLYTNVMDVjQWlFQXd2dWQKaTJnYVdYNDZhM2F2cHlWNW9iWWl3U25QbDFNNlJ6VWFJZldXSTNRPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCc3pDQ0FWcWdBd0lCQWdJUWVrVlpPMitUdG9RV1pJQkd4UDN2VXpBS0JnZ3Foa2pPUFFRREFqQWtNU0l3CklBWURWUVFERXhsVGRHVndJRU5sY25ScFptbGpZWFJsY3lCU2IyOTBJRU5CTUI0WERURTVNRGt5TXpFNE1qSXoKT0ZvWERUSTVNRGt5TURFNE1qSXpPRm93TERFcU1DZ0dBMVVFQXhNaFUzUmxjQ0JEWlhKMGFXWnBZMkYwWlhNZwpTVzUwWlhKdFpXUnBZWFJsSUVOQk1Ga3dFd1lIS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRU5kQmZVbCtZCnRMemNvU2cyMFl3UktmZzZ5dkRQT0tSRkl0UDhYeW1yOUtnSkprZW5MRmg5MEVsQmtSY3ZES3JPV1FPNldJZUgKQjd6VWlVbUtIUzFydTZObU1HUXdEZ1lEVlIwUEFRSC9CQVFEQWdFR01CSUdBMVVkRXdFQi93UUlNQVlCQWY4QwpBUUF3SFFZRFZSME9CQllFRklmWXdMbEgxTFJ1TEUrMUJ3RjBkS1gzNm1LVk1COEdBMVVkSXdRWU1CYUFGRVNUCkNDZkxMbW1IejZEb0hkU3djK2tFOEJiYk1Bb0dDQ3FHU000OUJBTUNBMGNBTUVRQ0lGRDR5SVRudm5lUEFlK1cKbHRibzd2T09wbW1IVXJKbnFXSm1GcktTOThKREFpQnNLLzd5SVZ2MEZ5Q0pnQmkzN0NNamIwb2NGTUhCb1NGUAo1Q28rZERrZEh3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBenlnREtmY2RzeDZYMXZGWHB1RktBbjBvTXFNc3p1eWZjdWVwd3Vvc3ZKR0dLQ2wzCksxMnI1RjJMcFMxSWJGOWxBR3pGQ0NjaG1PcDk4Rm90YjNjNTVURW1VT0lLV2ZsSnFZK2dDZEorUDRrSXpNaWYKWk8zWFg3MjlXNCtGZzdFbWdZSmtmVXlVb3pMQjNSNUtJb2lXdm5ROXAyNkk0eDMvcWZSeGNzK0hvc240SU5CYwo5d2Y2aTQ5TkpqVWZHSENWRWV6bE9EMDBuc1E1cHkvakFPaUZWYi9NclhadmNLWlA4SkdzUlJ5MEtqdnJHTHdhClk3Y1pqMHFGTkVZWmwxWVBMTDhPckhoazBKTDZ1aXZyUDJRSzQ1SGhhNkd4UHgxbUwrejJjSThPcEJGTTcrdTcKMXpndGlRR3RZMG1US3BqRFhWRVFIbzZaY3lZNnlMdzZDbTRYblFJREFRQUJBb0lCQVFDbldReU5SenR2NndRQQp1UUQxS0VTUWxDajRGMGZvZGRFdGt5Wlo5ckZqT0RCbm0yL1EwYUErSUsvaUpSMi9wcVRDWGh3aHhQTG9SRzZ1Cnp6cXJOR2JKMXhXNys0OGZOanFpeHdzalJDK0hCaDNjeVNteThFeXY1R2puWVo4YlR1M3JhN25nV3hBc2NxR1kKRHBNWEtMNldadk16b2lEbFVPT3JtWFJZcW54SkxXaXR3TjBkZmpQeUNQRlJhakNrZTVFTmJzTllFcUJ2ZVpmaQptb3UzY0ZuNGJORWJIa2pPajE3cTY0NUNIWXc0UExLTm94VDNudzNHL3RJVWtJOWZXc010eGFKbzRHOWFMSmdRCnhjMWJhMEV5UUpCcEs1VFVUYU4veG1aMkYyU2MzaVdrbzNabno3SDNTT0h0V1RJUUpQbGhHWG9JdkFTUExmaTIKc0ppSFJFRWhBb0dCQVBnN2NNbHpQalRZNnJGVWpUZDF4M01UL1p5VjNNS2ZCb0lkODduV0Y4V0tqTmRmbUZzMQp2eWFKNGU4UC9neXBWaDRvSWxpd2NWVmhaVWVrcEsvQ0Z2SllNS3JCcmF3aVQ1TFd5SGx3eUhsM3ZWaFpjY3BXClorQkdtODBBSDFJb3kwU0xwSUNVU0FDK3NlbmVGSEtBRm9IeTJCZzRYSWFIM1h1Lzk2ek42VDQ1QW9HQkFOV2oKaEg2OFE1UVRidnVFN2lodEpYdGRTMWRJRG9DWjgzL05hK3haQ1YxaWpyUExiUmgrNHo5aG90UG16VWhqT21XTgpnc0IwUmFydHUxZHAvUjhuMlUwSldzOXhyajZrYUxicWd2bTZrUjdpQmRWM2RhU084VkdIRW5TVloxcG1OUG5VCkhGRTBBWW80UW1jTlZUTGhHdUhPTHd5VEJaWS9UbUFJOWE3Ymh1U0ZBb0dBVmpyZnpUc1pTdVpuY0w1OFo5MmsKTVZlcWEvQ3JSWGhVbkhLbkxkRXZaTkhpWnhhd0tFOGRYZGZjQWwyOTFNd21QRWVZMXRKRUxIS3U2aHBKa3p5TApHQmlHbjJxMFYrMTdVdmtGbnhvTXFoVnNVdnFQU3NpRlFTV0dEY3FURnNsUkF6aTNKbXRjTDlZT203MG53VGE1Cm10NnZMb3ExcVk0YzVGWWxSWVFxR0JFQ2dZQTFiMWpNZWpwTTV5OXpLYVNINkFUZi9rM0IzdndCbm9BUkQwaFgKTFUzeVZaeHJ6QTNWZXFkTVN4ZDIrTnVQcDRJTHRJRm5zYTFiTzJzSVNqSk5nS0lSYXEyVUJWTVh4L2VKQ004Sgp0MmhrMlJPc2VOeXI5RFRYdDNrZVArQkI5QjU3L2xtMjFkTXN4eldVb1BrV2FNZFlnMEd4VzNTaVIrTktrZk1oCmNUY2tJUUtCZ0ZpeXR1RFNNLzdIdE41VE5kd0NPOTM2MjM0Y3MzQnhYZkpFSUNMVVcraXQ3bVZWSTlPRXhMK3IKK1c3L2pOZ2c5VVBPKzZZdVRvN2xQemVTbVVMWFB6bXNTRVdaQnI5M2lLZUR5TE0rK094NlB3blZ0ZzJDcnRObQpob3RsbmN2bHB6OWpuVXRNSS9rVm1Wdm9RZmhsQ2FuenAvTWJocVZhdlNxSGlwUWxOY0tMCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
kind: Secret
metadata:
annotations:
certmanager.k8s.io/alt-names: test.smallstep.com,localhost
certmanager.k8s.io/certificate-name: test-smallstep-com
certmanager.k8s.io/common-name: test.smallstep.com
certmanager.k8s.io/ip-sans: 127.0.0.1
certmanager.k8s.io/issuer-kind: CertificateRequest
certmanager.k8s.io/issuer-name: step-issuer
creationTimestamp: "2019-09-23T19:14:24Z"
name: test-smallstep-com-tls
namespace: default
resourceVersion: "23247"
selfLink: /api/v1/namespaces/default/secrets/test-smallstep-com-tls
uid: 611663e6-51ae-4d4f-a4ff-1d5fd1baf74b
type: kubernetes.io/tls@devops2201 looking at cert-manager code base, the error in your log appears when the certificate request reason is not failed or issued https://github.com/jetstack/cert-manager/blob/abb680756e84b56df47f85a8ce732a0e09e0f171/pkg/controller/certificates/sync.go#L482
Can you show the output of kubectl get certificaterequests <the-resource-name> -o yaml?
-> I had tried this earlier by not enabling the argument that you suggested but I can give a retry now that I have enabled the argument.
CertificateRequest resources should work without the flag
With CertificateRequest, I will have to create a CSR and base64 encode it and then pass it in the YAML definition.
You can use step-issuer/config/sample/certificaterequest.yaml for example.
kc get certificaterequest xxx-o yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: CertificateRequest
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"CertificateRequest","metadata":{"annotations":{},"name":"xxxdev-net","namespace":"default"},"spec":{"csr":"xxx","duration":"336h","isCA":false,"issuerRef":{"group":"certmanager.step.sm","name":"step-issuer"}}}
creationTimestamp: "2019-09-23T19:31:43Z"
generation: 2
name: xxx-net
namespace: default
resourceVersion: "1877244"
selfLink: /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificaterequests/xxxdev-net
uid: c652c939-de38-11e9-8674-d685fc183866
spec:
csr: xxx
duration: 336h0m0s
issuerRef:
group: certmanager.step.sm
name: step-issuer
status:
conditions:
- lastTransitionTime: "2019-09-23T19:31:43Z"
message: 'Failed to sign certificate request: Unauthorized'
reason: Failed
status: "False"
type: Ready
Ok, that Unauthorized is a response of step-certificates, in step-certificates log you should be able to see a more descriptive error, but my current guess is that the step-issuer is not properly configured.
@devops2201 If you want, you can ping me on https://gitter.im/smallstep/community and we can dig further.
I am amused how a secret with TLS cert is getting created
kc describe secret xxx-tls
Name: xxx-tls
Namespace: default
Labels:
Annotations: certmanager.k8s.io/alt-names: *.xxxdev.net
certmanager.k8s.io/certificate-name: xxxx
certmanager.k8s.io/common-name: *.xxxdev.net
certmanager.k8s.io/ip-sans:
certmanager.k8s.io/issuer-kind: CertificateRequest
certmanager.k8s.io/issuer-name: step-issuer
Type: kubernetes.io/tls
Data
ca.crt: 0 bytes
tls.crt: 993 bytes
tls.key: 1679 bytes
Sure I will reach out to you on that link @maraino
Not an issue.