jx-build-controller after admin operator install results in CrashLoopBackOff status
Closed this issue · 1 comments
adgantt commented
jx version: version: 3.2.188
Our team is trying to setup JenkinsX through the admin operator on a vanilla Kubernetes cluster in AWS EKS. Using a self hosted Gitlab repository and not using public Github.com.
Created our repository in our Gitlab instance with the following command:
jx admin create --git-kind gitlab --git-name adgantt --git-server https://gitlab.team-domain.com --git-token <PAT> --git-public false --no-operator --repo jenkins-x-admin
Repository creation was successful and looks like the expected files are present in the repo.
Proceeded to run the following admin operator command:
jx admin operator --username adgantt --token <PAT> -u https://gitlab.team-domain.com/admins/jenkins-x-admin
The installation runs as expected and logs the following health status summary:
jx gitops webhook update --warn-on-fail
WARNING: failed to create Scm client for https://github.com/admins/jenkins-x-admin.git: failed to find git token: could not find git token for git server https://github.com
jx health status -A
NAME NAMESPACE STATUS ERROR MESSAGE
daemonset kuberhealthy OK
deployment kuberhealthy OK
dns-status-internal kuberhealthy OK
jx-bot-token jx ERROR Check execution error: pod deleted unexpectedly
jx-pod-status kuberhealthy OK
jx-secrets kuberhealthy OK
jx-webhook jx ERROR Check execution error: jx/jx-webhook: error when waiting for pod to start: the khcheck check pod is deleted, waiting for start failed
jx-webhook-events jx ERROR Check execution error: jx/jx-webhook-events: error when waiting for pod to start: the khcheck check pod is deleted, waiting for start failed
network-connection-check kuberhealthy OK
Please note in the output, the reference to github.com instead of our expected gitlab.team-domain.com.
When reviewing the namespaces created from the process:
% kubectl get namespaces
NAME STATUS AGE
default Active 7d6h
istio-system Active 2d5h
jx Active 4h2m
jx-git-operator Active 7d4h
jx-production Active 4h2m
jx-staging Active 4h2m
kube-node-lease Active 7d6h
kube-public Active 7d6h
kube-system Active 7d6h
kuberhealthy Active 4h2m
nginx Active 4h2m
tekton-pipelines Active 4h2m
Looking at the pods in the various namespaces, I see the following issue in the jx namespace:
% kubectl -n jx get pods
NAME READY STATUS RESTARTS AGE
bucketrepo-bucketrepo-5697f998bc-pb2w7 1/1 Running 0 4h29m
jx-build-controller-57b8fcdb55-pthlv 0/1 CrashLoopBackOff 57 4h29m
jx-gcactivities-27175320-thlnj 0/1 Completed 0 61m
jx-gcactivities-27175350-fb6rj 0/1 Completed 0 31m
jx-gcpods-27175320-dkthf 0/1 Completed 0 61m
jx-gcpods-27175350-tn565 0/1 Completed 0 31m
jx-pipelines-visualizer-779cc99c9-xnkkd 1/1 Running 0 4h25m
jx-preview-gc-jobs-27175360-klqwd 0/1 Completed 0 21m
jx-preview-gc-jobs-27175370-gwngg 0/1 Completed 0 11m
jx-preview-gc-jobs-27175380-6q6mp 0/1 Completed 0 65s
lighthouse-foghorn-6b4dcfb95d-c4qdt 1/1 Running 0 4h25m
lighthouse-gc-jobs-27175320-s99qr 0/1 Completed 0 61m
lighthouse-gc-jobs-27175350-vl4fq 0/1 Completed 0 31m
lighthouse-gc-jobs-27175380-hfckw 0/1 Completed 0 65s
lighthouse-keeper-8579c6c87-4qzsw 1/1 Running 0 4h25m
lighthouse-tekton-controller-684b96dd8d-t2h6c 1/1 Running 0 4h25m
lighthouse-webhooks-5d8fbd6fbf-rbbbd 1/1 Running 0 35m
The logs for the jx-build-controller pod show the following, which also notes references to github.com instead of our expected Gitlab domain of gitlab.team-domain.com
% kubectl logs -n jx jx-build-controller-57b8fcdb55-pthlv
{"endpoint":"tempo.jx-observability:14268","level":"info","msg":"Initializing OpenTelemetry Traces Exporter","time":"2021-09-01T18:59:49Z","type":"jaeger:http:thrift"}
{"error":"Post \"http://tempo.jx-observability:14268/api/traces\": dial tcp: lookup tempo.jx-observability on 10.100.0.10:53: no such host","level":"warning","msg":"Traces Exporter Endpoint configuration error. Maybe you need to install/configure the Observability stack? https://jenkins-x.io/v3/admin/guides/observability/ The OpenTelemetry Tracing feature won't be enabled until this is fixed.","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"about to run: git config --global credential.helper store in dir .","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"starting build controller","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"starting secret watching masker","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"Watching for Secret resources in namespace jx","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"Watching for Secret resources in namespace jx-git-operator","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"Watching for PipelineActivity resources in namespace jx","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"The service is ready to listen and serve.","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"ignoring secret bucketrepo-config","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"adding mask of secret jenkins-maven-settings entry settings-security.xml ","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"adding mask of secret jenkins-maven-settings entry settings.xml ","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"ignoring secret jenkins-x-bucketrepo entry BASIC_AUTH_USER","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"adding mask of secret jenkins-x-bucketrepo entry BASIC_AUTH_PASS ","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"adding mask of secret jx-basic-auth-htpasswd entry auth ","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"adding mask of secret jx-basic-auth-user-password entry password ","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"adding mask of secret jx-basic-auth-user-password entry username ","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"ignoring secret jx-boot entry username","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"adding mask of secret jx-boot entry gitInitCommands ","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"adding mask of secret jx-boot entry password ","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"ignoring secret jx-boot entry url","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"adding mask of secret lighthouse-hmac-token entry hmac ","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"adding mask of secret lighthouse-oauth-token entry oauth ","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"adding mask of secret tekton-container-registry-auth entry .dockerconfigjson ","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"adding mask of secret tekton-git entry username ","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"adding mask of secret tekton-git entry password ","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"wrote git credentials file /tmp/git-creds/git/credentials","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"about to run: git clone https://github.com/admins/jenkins-x-admin.git /tmp/jx-git-154599549 in dir /tmp","time":"2021-09-01T18:59:49Z"}
{"level":"info","msg":"Cloning into '/tmp/jx-git-154599549'...\nremote: Support for password authentication was removed on August 13, 2021. Please use a personal access token instead.\nremote: Please see https://github.blog/2020-12-15-token-authentication-requirements-for-git-operations/ for more information.\nfatal: Authentication failed for 'https://github.com/admins/jenkins-x-admin.git/'","time":"2021-09-01T18:59:50Z"}
{"level":"fatal","msg":"failed to get cluster requirements: failed to clone cluster git repo https://github.com/admins/jenkins-x-admin.git: failed to clone repository https://github.com/admins/jenkins-x-admin.git to directory: /tmp/jx-git-154599549: failed to run 'git clone https://github.com/admins/jenkins-x-admin.git /tmp/jx-git-154599549' command in directory '/tmp', output: 'Cloning into '/tmp/jx-git-154599549'...\nremote: Support for password authentication was removed on August 13, 2021. Please use a personal access token instead.\nremote: Please see https://github.blog/2020-12-15-token-authentication-requirements-for-git-operations/ for more information.\nfatal: Authentication failed for 'https://github.com/admins/jenkins-x-admin.git/''","time":"2021-09-01T18:59:50Z"}
It appears that our non-Github.com git URL is not making its way through each of the propagated commands. We could use some help resolving the issue to use our Gitlab repository instead of the default Github.com service.
adgantt commented
Found an update in the jx-requirements.yml file to specify the Gitlab server, and this seems to correct the issue with the jx-build-controller pod issue above. Closing the ticket as no longer an issue.
For reference, Git configuration reference: https://jenkins-x.io/v3/admin/setup/config/git/