kube-lego tls recursion
martell opened this issue · 32 comments
I am having issues with tls integration with kube-lego following the blog.
ACME account registration is successful.
and I have config.LEGO_URL=https://acme-v01.api.letsencrypt.org/directory for production
level=warning msg="authorization failed after 1m0s: reachability test failed: Get https://gitlab.mydomain.com/users/sign_in: dial tcp x.x.x.x:443: getsockopt: connection refused" context=acme domain=gitlab.mydomain.com
It seems as though the gitlab redirect is failing because it has no tls cert.
kube-lego seems to try reaching https://gitlab.mydomain.com which fails because of the redirect.
Did you say recursion? :)
I seeing a lot of problems with kube-lego with on RBAC-enabled setups.
Does your kube-lego work (I mean with any other domains)?
Btw, did you edit posted log manually? It has two domains in one line: https://gitlab.marineci.com and gitlab.mydomain.com.
I seeing a lot of problems with kube-lego with on RBAC-enabled setups.
Does your kube-lego work (I mean with any other domains)?
Yes it works with a simple nginx https server
I had this issue even without rbac.
Btw, did you edit posted log manually?
I manually edited the log yes. Edited them both now
Hmmm. Took another look at your first comment. It seems that you have a problem with ingress/ingress-controller and not kube-lego.
dial tcp x.x.x.x:443: getsockopt: connection refused" - you should be able to reach this url/IP even without certificates
hmm
kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
austere-spaniel-gitlab gitlab.mydomain.com 80, 443 2h
I can curl the ip on port 80
curl x.x.x.x:80
<html><body>You are being <a href="https://x.x.x.x/users/sign_in">redirected</a>.</body></html>%
but not 433 even though gitlab should allow that.
svc/austere-spaniel-gitlab 10.0.25.52 x.x.x.x 22:30834/TCP,80:30495/TCP,443:31166/TCP
Can you check your config for this settings?
GITLAB_HTTPS: "true"
GITLAB_PORT: "443"
GITLAB_PORT: 443
GITLAB_HTTPS: "true"
I have them both already.
Will attach the full config
I am using 9.3.7 and RELEASE.2017-06-13T19-01-01Z.
Using the older ones does not change this in anyway.
Could you please mark lines that you intentionally edited with something like "" or "". It's really hard to understand now.
For example, ingress rules are for gitlab.example.com which is the same as in my examples, but it should be like that.
used mydomain.com and ""x"" where username / passwords were changed
## GitLab image
## ref: https://hub.docker.com/r/sameersbn/gitlab/
##
image: sameersbn/gitlab
## Gitlab image version
## ref: https://hub.docker.com/r/sameersbn/gitlab/tags/
##
imageTag: "9.3.7"
## Specify a imagePullPolicy
## 'Always' if imageTag is 'latest', else set to 'IfNotPresent'
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
##
# imagePullPolicy:
## For minikube, set this to NodePort, elsewhere use LoadBalancer
## ref: http://kubernetes.io/docs/user-guide/services/#publishing-services---service-types
##
serviceType: LoadBalancer
## Configure external service ports
## ref: http://kubernetes.io/docs/user-guide/services/
sshPort: 22
httpPort: 80
httpsPort: 443
ingress:
enabled: true
## Gitlab Ingress annotations
##
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: 'true'
## Gitlab Ingress hostnames
## Must be provided if Ingress is enabled
##
hosts:
- gitlab.mydomain.com
## Gitlab Ingress TLS configuration
## Secrets must be manually created in the namespace
##
tls:
- secretName: gitlab-server-tls
hosts:
- gitlab.mydomain.com
## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
## GitLab requires a good deal of resources. We have split out Postgres and
## redis, which helps some. Refer to the guidelines for larger installs.
## ref: https://docs.gitlab.com/ce/install/requirements.html#hardware-requirements
requests:
memory: 1Gi
cpu: 500m
limits:
memory: 2Gi
cpu: 1
## Enable persistence using Persistent Volume Claims
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
## ref: https://docs.gitlab.com/ce/install/requirements.html#storage
##
persistence:
## This volume persists generated configuration files, keys, and certs.
##
enabled: true
size: 10Gi
## If defined, volume.beta.kubernetes.io/storage-class: <storageClass>
## Default: volume.alpha.kubernetes.io/storage-class: default
##
# storageClass: ""
accessMode: ReadWriteOnce
## Configuration values for the postgresql dependency.
## ref: https://github.com/kubernetes/charts/blob/master/stable/postgresql/README.md
##
postgresql:
# 9.6 is the newest supported version for the GitLab container
imageTag: "9.6"
cpu: 1000m
memory: 1Gi
postgresUser: ""x""
postgresPassword: ""x""
postgresDatabase: ""x""
persistence:
size: 10Gi
## Configuration values for the redis dependency.
## ref: https://github.com/kubernetes/charts/blob/master/stable/redis/README.md
##
redis:
image: redis:3.2.8
redisPassword: ""x""
resources:
requests:
memory: 1Gi
persistence:
size: 10Gi
minio:
# https://hub.docker.com/r/minio/minio/tags/
imageTag: "RELEASE.2017-06-13T19-01-01Z"
mode: "standalone"
accessKey: ""x""
secretKey: ""x""
persistence:
enabled: true
size: 20Gi
config:
DEBUG: "false"
GITLAB_ROOT_EMAIL: ""x""
GITLAB_ROOT_PASSWORD: ""x""
GITLAB_HOST: gitlab.mydomain.com
GITLAB_PORT: 443
GITLAB_HTTPS: "true"
GITLAB_SSH_PORT: "22"
GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN: ""x""
runner:
enabled: true
gitlabUrl: "https://gitlab.mydomain.com/ci/"
registrationToken: ""x""
cache:
enabled: true
accessKey: ""x""
secretKey: "x"
registry:
enabled: false
Config looks fine.
Let's double check kube-lego...
you should have ingress in kube-system (or where your kube-lego is deployed) with entry for gitlab.
something like this:
$ kubectl get ing kube-lego-nginx --namespace=kube-system -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme-challenge-endpoints: "true"
name: kube-lego-nginx
namespace: kube-system
spec:
rules:
- host: gitlab.yourdomain.com
http:
paths:
- backend:
serviceName: kube-lego-nginx
servicePort: 8080
path: /.well-known/acme-challenge
and service kube-lego-nginx in the same namespace.
Also what does your ingress-controller logs say when you see this errors in kube-lego?
The ingress and service are in the default namespace
kubectl get ing kube-lego-nginx -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme-challenge-endpoints: "true"
creationTimestamp: ""x""
generation: 1
name: kube-lego-nginx
namespace: default
resourceVersion: "8432"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/kube-lego-nginx
uid: ""x""
spec:
rules:
- host: gitlab.mydomain.com
http:
paths:
- backend:
serviceName: kube-lego-nginx
servicePort: 8080
path: /.well-known/acme-challenge
status:
loadBalancer: {}
kubectl get svc kube-lego-nginx -o yaml
apiVersion: v1
kind: Service
metadata:
annotations:
kubernetes.io/kube-lego-managed: "true"
creationTimestamp: ""x""
name: kube-lego-nginx
namespace: default
resourceVersion: "8431"
selfLink: /api/v1/namespaces/default/services/kube-lego-nginx
uid: ""x""
spec:
clusterIP: ""10.0.x.x""
ports:
- port: 8080
protocol: TCP
targetPort: 8080
selector:
app: kube-lego
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
Also what does your ingress-controller logs say when you see this errors in kube-lego?
I am not quite sure how I view the logs of the ingress-controller.
kubectl logs ing/kube-lego-nginx
error: cannot get the logs from extensions/__internal, Kind=Ingress
Nevermind my last question, I assumed that you're using ingress-controller. something like nginx-ingress
Okay, let's try this:
does curl http://gitlab.mydomain.com/.well-known/acme-challenge/_selftest work?
<html><body>You are being <a href="https://gitlab.mydomain.com/users/sign_in">redirected</a>.</body></html>
It seems like your LoadBalancer does not allow http, and forces https.
How did you configure your LoadBalancer rules in Azure?
Done a little bit of reading.
There are 4 rules one for each of these ports 22, 80, 443 and 9000.
The backend port matches the frontend port in each rule.
It seems ssh, http, https and minio should all be in order.
I always add ingress-controller. But in the cloud it should work even without it (assuming configured load balancer).
You should find who forces the redirect for curl http://gitlab.mydomain.com/.well-known/acme-challenge/_selftest
Hey @lwolf a bit of an update here.
I managed to resolve the tls issue by switching to jetstack-experimental/cert-manager and resolving the acme challenge at the dns level.
The secret with the public and private key is now valid.
However the website is still only accessible via http and not https.
curl https://gitlab.example.com`
curl: (7) Failed to connect to gitlab.example.com port 443: Connection refused
curl gitlab.example.com
<html><body>You are being <a href="https://gitlab.example.com/users/sign_in">redirected</a>.</body></html>
curl https://gitlab..example.com/users/sign_in
curl: (7) Failed to connect to gitlab.example.com port 443: Connection refused
Is there something I am missing to open the port because kubectl get shows the following
svc/luminous-butterfly-gitlab 10.0.x.x x.x.x.x 22:32536/TCP,80:32418/TCP,443:31610/TCP 28m
Am I supposed to edit the service to remove http support and remap https to the nodePort of http?
After a lot more digging I found
https://github.com/sameersbn/docker-gitlab#installation-of-the-ssl-certificates
It seems the default for SSL_CERTIFICATE_PATH and SSL_KEY_PATH do not exist.
/home/git/data/certs/gitlab.key or /home/git/data/certs/gitlab.crt
/home/git/data/ssh/ seems to have a bunch of files like ssh_host_dsa_key ssh_host_dsa_key.pub ssh_host_ecdsa_key ssh_host_ecdsa_key.pub ssh_host_ed25519_key ssh_host_ed25519_key.pub ssh_host_key ssh_host_key.pub ssh_host_rsa_key ssh_host_rsa_key.pub however.
Is there some mistake in the helm chart where it just maps these variables instead of pulling the key from the secret ?
Am I supposed to edit the service to remove http support and remap https to the nodePort of http?
No you do not need to do any of this.
It seems the default for SSL_CERTIFICATE_PATH and SSL_KEY_PATH do not exist.
You don't have to have SSL certificates inside the gitlab since you're doing the termination at the ingress level, so that's fine.
Could you check access to gitlab using NodePorts? something like this:
curl -k -H "Host: gitlab.yourdomain" https://any-node-ip-address:https-node-port/
curl -k -H "Host: gitlab.example.com" https://10.0.29.101:31610/
hangs where 10.0.29.101 is the cluster ip and 443 is mapped to 31610
curl -k -H "Host: gitlab.example.com" https://10.0.29.105:433/
curl: (7) Failed to connect to 104.40.191.52 port 443: Connection refused
and the external ip does the same.
You should try Node IPaddress with nodePort, not clusterIP.
for example: curl -k -H "Host: gitlab.example.com" https://10.0.29.105:31610/
The service type is LoadBalancer though not NodePort.
Name: luminous-butterfly-gitlab
Namespace: default
Labels: app=luminous-butterfly-gitlab
chart=gitlab-0.2.8
heritage=Tiller
release=luminous-butterfly
Annotations: <none>
Selector: app=luminous-butterfly-gitlab
Type: LoadBalancer
IP: 10.0.29.101
LoadBalancer Ingress: x.x.x.x
Port: ssh 22/TCP
NodePort: ssh 32536/TCP
Endpoints: 10.244.0.17:22
Port: https 443/TCP
NodePort: https 31610/TCP
Endpoints: 10.244.0.17:443
Port: http 80/TCP
NodePort: http 32418/TCP
Endpoints: 10.244.0.17:80
Session Affinity: None
Events: <none>
curl -k -H "Host: gitlab.example.com" https://10.244.0.17:433/
curl: (7) Failed to connect to 10.244.0.17 port 433: Connection refused
curl -k -H "Host: gitlab.example.io" https://10.244.0.17:31610/
curl: (7) Failed to connect to 10.244.0.17 port 31610: Connection refused
I believe this is what you want?
Just to clarify.
10.244.0.0/24 - POD network
10.0.29.101 - your physical node IP
If that is the case, commands should be like this:
curl -k -H "Host: gitlab.example.io" https://10.0.29.101:31610/ - to test using NodePort
curl -k -H "Host: gitlab.example.com" https://10.244.0.17:433/ - to test from inside the cluster
But anyway, it clearly does not work.
Could you check that port 443 is actually in use inside the gitlab pod
$ kubectl exec -it gitlab-pod -- netstat -anp | grep 443
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 3293/nginx -g daemo
The grep doesn't return anything so it doesn't look like it
great, now we know that it is something with your gitlab config.
Just need to figure out why your gitlab only serves HTTP :)
I thought the gitlab config was generated from the helm chart?
I have
GITLAB_HOST: gitlab.mydomain.com
GITLAB_PORT: 443
GITLAB_HTTPS: "true"
exec'ing bash in the pod and printing out the env vars reflect this.
The only difference from above is I have the runner and registry disabled.
Yes it is generated by chart from the values you provide.
And the values looks fine.
Could you show me the ingress output?
$kubectl get ing gitlab-ingress-name -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
creationTimestamp: 2017-09-04T17:51:45Z
generation: 1
labels:
app: luminous-butterfly-gitlab
chart: gitlab-0.2.8
heritage: Tiller
release: luminous-butterfly
name: luminous-butterfly-gitlab
namespace: default
resourceVersion: "11557"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/luminous-butterfly-gitlab
uid: b7b0b276-9199-11e7-b869-000d3a2bfde3
spec:
rules:
- host: gitlab.example.com
http:
paths:
- backend:
serviceName: luminous-butterfly-gitlab
servicePort: 80
path: /
- http:
paths:
- backend:
serviceName: luminous-butterfly-gitlab
servicePort: 22
path: /
tls:
- hosts:
- gitlab.example.com
secretName: example-com
status:
loadBalancer: {}
It doesn't have 443 but kubectl get ing shows it does.
I added the following for testing
- http:
paths:
- backend:
serviceName: luminous-butterfly-gitlab
servicePort: 443
path: /
Still nothing on curl but I assume that is to be expected.
Yes, everything is correct.
Another thing I noticed, you have secretName - example-com in this ingress, but in Values.yaml in the beginning, it says gitlab-server-tls. Did you do something to it?
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: 'true'
hosts:
- gitlab.mydomain.com
tls:
- secretName: gitlab-server-tls
hosts:
- gitlab.mydomain.com
Also
Did you look at logs in gitlab pod during the start?
/var/log/gitlab/gitlab/production.log
Other logs in gitlab folder could also be useful.
That was not the issue either, that was just an edited value.
I managed to get this working by setting up an ingress controller on the kube-lego namespace.
I used nginx-ingress-controller.yaml from kubetnetes/ingress and created a configmap for port 22 to a higher port number and back because you can't bind 22 all the way though.
I am going to create an updated helm chart at some point for creating an ingress controller as part of the gitlab setup with a configmap for 433 80 and 22.
Thanks for the help I just bought you a beer :)
weird stuff.. anyway I'm glad that it works now.
thanks :)