Can't startup registry with Azure backend
monaka opened this issue · 5 comments
monaka commented
I'm trying to install Deis with Azure Blob but failed.
I re-generated from chart, uninstalled, installed after editing ~/.helmc/workspace/charts/workflow-v2.0.0/tpl/generate_params.toml
The log is like this:
$ kubectl logs -f deis-registry-34xc6 --namespace=deis
2016/06/20 00:42:05 INFO: Starting registry...
2016/06/20 00:42:05 INFO: using azure as the backend
But it can't be started up....
$ kubectl get pods --namespace=deis
(snip)
deis-registry-34xc6 0/1 CrashLoopBackOff 11 20m
For checking my account information, I run the docker registry on my node manually with my info.
It seems this works well.
$ docker run -it --rm -p 5000:5000 -e REGISTRY_STORAGE=azure -e REGISTRY_STORAGE_AZURE_ACCOUNTNAME="{account_name_censored}" -e
REGISTRY_STORAGE_AZURE_ACCOUNTKEY="{account_key_censored}" -e REGISTRY_STORAGE
_AZURE_CONTAINER="registry" --name=registry registry:2
WARN[0000] No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable. go.version=go1.6.2 instance.id=e1485474-39bf-4589-8939-9eb6d5385ef6 version=v2.4.1
INFO[0000] redis not configured go.version=go1.6.2 instance.id=e1485474-39bf-4589-8939-9eb6d5385ef6 version=v2.4.1
INFO[0000] Starting upload purge in 23m0s go.version=go1.6.2 instance.id=e1485474-39bf-4589-8939-9eb6d5385ef6 version=v2.4.1
INFO[0000] listening on [::]:5000 go.version=go1.6.2 instance.id=e1485474-39bf-4589-8939-9eb6d5385ef6 version=v2.4.1
I think there have some troubles in /bin/registry. But I can't determine as there have few logs.
Could you tell me some tips for shooting this kind of trouble?
kmala commented
i had just tried using azure on workflow-v2.0.0 chart and it seems to be working....can you say what kubectl describe pod <registry-pod> --namespace=deis say and also can you get the logs of the container by deleting the pod and before it restarting/going into crashloop because sometime kubectl misses some logs when the container is in crashloopbackoff.
monaka commented
The result of describe pods like this.
Seems registry daemon didn't spawn...
$ kubectl describe pods deis-registry-e3ztm --namespace deis
Name: deis-registry-e3ztm
Namespace: deis
Node: 10.0.2.4/10.0.2.4
Start Time: Mon, 20 Jun 2016 03:03:02 +0000
Labels: app=deis-registry
Status: Running
IP: 172.16.43.10
Controllers: ReplicationController/deis-registry
Containers:
deis-registry:
Container ID: docker://960a9c38f1d365667ebc5061a7106a73286fe28c4efdcf651feef2ef40c82542
Image: quay.io/deis/registry:v2.0.0
Image ID: docker://sha256:4d0dfeacd8e622bee7446ff905b9c76ff5c2c94bfaaceef094e0a52296b0d098
Port: 5000/TCP
QoS Tier:
memory: BestEffort
cpu: BestEffort
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Mon, 20 Jun 2016 03:04:12 +0000
Finished: Mon, 20 Jun 2016 03:04:22 +0000
Ready: False
Restart Count: 3
Liveness: http-get http://:5000/v2/ delay=1s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:5000/v2/ delay=1s timeout=1s period=10s #success=1 #failure=3
Environment Variables:
REGISTRY_STORAGE_DELETE_ENABLED: true
REGISTRY_LOG_LEVEL: info
REGISTRY_STORAGE: azure
Conditions:
Type Status
Ready False
Volumes:
registry-storage:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
registry-creds:
Type: Secret (a volume populated by a Secret)
SecretName: objectstorage-keyfile
deis-registry-token-p6m3l:
Type: Secret (a volume populated by a Secret)
SecretName: deis-registry-token-p6m3l
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1m 1m 1 {default-scheduler } Normal Scheduled Successfully assigned deis-registry-e3ztm to 10.0.2.4
1m 1m 1 {kubelet 10.0.2.4} spec.containers{deis-registry} Normal Created Created container with docker id e6b44596f083
1m 1m 1 {kubelet 10.0.2.4} spec.containers{deis-registry} Normal Started Started container with docker id e6b44596f083
32s 32s 1 {kubelet 10.0.2.4} spec.containers{deis-registry} Normal Killing Killing container with docker id e6b44596f083: pod "deis-registry-e3ztm_deis(7da1c1b4-3693-11e6-9181-000d3a50375f)" container "deis-registry" is unhealthy, it will be killed and re-created.
32s 32s 1 {kubelet 10.0.2.4} spec.containers{deis-registry} Normal Started Started container with docker id 23e5fde34cb5
32s 32s 1 {kubelet 10.0.2.4} spec.containers{deis-registry} Normal Created Created container with docker id 23e5fde34cb5
22s 22s 1 {kubelet 10.0.2.4} spec.containers{deis-registry} Normal Killing Killing container with docker id 23e5fde34cb5: pod "deis-registry-e3ztm_deis(7da1c1b4-3693-11e6-9181-000d3a50375f)" container "deis-registry" is unhealthy, it will be killed and re-created.
22s 22s 1 {kubelet 10.0.2.4} spec.containers{deis-registry} Normal Created Created container with docker id 1bd56ff56fa2
22s 22s 1 {kubelet 10.0.2.4} spec.containers{deis-registry} Normal Started Started container with docker id 1bd56ff56fa2
1m 12s 4 {kubelet 10.0.2.4} spec.containers{deis-registry} Normal Pulled Container image "quay.io/deis/registry:v2.0.0" already present on machine
12s 12s 1 {kubelet 10.0.2.4} spec.containers{deis-registry} Normal Killing Killing container with docker id 1bd56ff56fa2: pod "deis-registry-e3ztm_deis(7da1c1b4-3693-11e6-9181-000d3a50375f)" container "deis-registry" is unhealthy, it will be killed and re-created.
12s 12s 1 {kubelet 10.0.2.4} spec.containers{deis-registry} Normal Created Created container with docker id 960a9c38f1d3
12s 12s 1 {kubelet 10.0.2.4} spec.containers{deis-registry} Normal Started Started container with docker id 960a9c38f1d3
52s 2s 6 {kubelet 10.0.2.4} spec.containers{deis-registry} Warning Unhealthy Readiness probe failed: Get http://172.16.43.10:5000/v2/: dial tcp 172.16.43.10:5000: connection refused
52s 2s 6 {kubelet 10.0.2.4} spec.containers{deis-registry} Warning Unhealthy Liveness probe failed: Get http://172.16.43.10:5000/v2/: dial tcp 172.16.43.10:5000: connection refused
2s 2s 1 {kubelet 10.0.2.4} spec.containers{deis-registry} Normal Killing Killing container with docker id 960a9c38f1d3: pod "deis-registry-e3ztm_deis(7da1c1b4-3693-11e6-9181-000d3a50375f)" container "deis-registry" is unhealthy, it will be killed and re-created.
2s 1s 2 {kubelet 10.0.2.4} spec.containers{deis-registry} Warning BackOff Back-off restarting failed docker container
2s 1s 2 {kubelet 10.0.2.4} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "deis-registry" with CrashLoopBackOff: "Back-off 20s restarting failed container=deis-registry pod=deis-registry-e3ztm_deis(7da1c1b4-3693-11e6-9181-000d3a50375f)"
kmala commented
yes..it seems so....can you delete registry pod and get the logs immediatly because sometime the kubectl logs doesn't show full logs when its in crashloopback
monaka commented
I'm keeping inspection. Possibly it is caused lower layer than Registry.
monaka commented
It's my mistake. DNS service in Kube-system was down. After fixing it, Registry was up.
I close this issue.
2016/06/20 04:53:56 INFO: Starting registry...
2016/06/20 04:53:56 INFO: using azure as the backend
time="2016-06-20T04:53:57.120359633Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.6.1 instance.id=bc90833f-d224-44d0-bad5-d57d6e1856af service=registry version=v2.1.1-497-g5e84d4f
time="2016-06-20T04:53:57.120443037Z" level=info msg="redis not configured" go.version=go1.6.1 instance.id=bc90833f-d224-44d0-bad5-d57d6e1856af service=registry version=v2.1.1-497-g5e84d4f
(snip)