deis/registry

Can't connect to registry pod

arkkanoid opened this issue · 4 comments

arkkanoid commented

Hi,
I'm building apps to Deis V2 with S3 backend. When i run "git push deis master" I get this error:

Removing intermediate container aafad5c94f80
Successfully built 972c4c508b78
pushing to registry
remote: ":"The push refers to a repository [10.0.1.20:80/prova9] (len: 1)"}

{"errorDetail":{"message":"unable to ping registry endpoint https://10.0.1.20:80/v0/\nv2 ping attempt failed with error: Get https://10.0.1.20:80/v2/: tls: oversized record     received with length 20527\n v1 ping attempt failed with error: Get https://10.0.1.20:80/v1/_ping: tls: oversized record received with length 20527"},"error":"unable to ping     registry endpoint https://10.0.1.20:80/v0/\nv2 ping attempt failed with error: Get https://10.0.1.20:80/v2/: tls: oversized record received with length 20527\n v1 ping attempt     failed with error: Get https://10.0.1.20:80/v1/_ping: tls: oversized record received with lengtremote: }

Build complete.
Launching app.

How can I reestablish the Registry component?

kubectl describe pod/prova9-v2-cmd-v1jhr --namespace=prova9

  2m    2m    1 {default-scheduler }                  Scheduled Successfully assigned prova9-v2-cmd-v1jhr to ip-172-20-0-67.eu-west-1.compute.internal
  2m    48s   4 {kubelet ip-172-20-0-67.eu-west-1.compute.internal} spec.containers{prova9-cmd} Pulling   pulling image "10.0.1.20:80/prova9:git-de15cce0"
  2m    48s   4 {kubelet ip-172-20-0-67.eu-west-1.compute.internal} spec.containers{prova9-cmd} Failed    Failed to pull image "10.0.1.20:80/prova9:git-de15cce0": API error     (500): unable to ping registry endpoint https://10.0.1.20:80/v0/
v2 ping attempt failed with error: Get https://10.0.1.20:80/v2/: tls: oversized record received with length 20527
 v1 ping attempt failed with error: Get https://10.0.1.20:80/v1/_ping: tls: oversized record received with length 20527

  2m  48s 4 {kubelet ip-172-20-0-67.eu-west-1.compute.internal}   FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "prova9-cmd" with ErrImagePull: "API     error (500): unable to ping registry endpoint https://10.0.1.20:80/v0/\nv2 ping attempt failed with error: Get https://10.0.1.20:80/v2/: tls: oversized record received with     length 20527\n v1 ping attempt failed with error: Get https://10.0.1.20:80/v1/_ping: tls: oversized record received with length 20527\n"

  2m  3s  8 {kubelet ip-172-20-0-67.eu-west-1.compute.internal} spec.containers{prova9-cmd} BackOff   Back-off pulling image "10.0.1.20:80/prova9:git-de15cce0"
  2m  3s  8 {kubelet ip-172-20-0-67.eu-west-1.compute.internal}         FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "prova9-cmd" with     ImagePullBackOff: "Back-off pulling image \"10.0.1.20:80/prova9:git-de15cce0\""

In controller container I run:
telnet 10.0.1.20 80
HTTP/1.1 400 Bad Request

In registry container I run:
telnet 127.0.0.1 80
telnet: can't connect to remote host (127.0.0.1): Connection refused

Registry logs:

...
2016-04-12T08:05:52.707644464Z time="2016-04-12T08:05:52.674220195Z" level=info msg="response completed" go.version=go1.5.3 http.request.host="10.244.0.6:5000" http.request.    id=3a83f284-16c0-44e9-87a2-be0e27ff2b6e http.request.method=GET http.request.remoteaddr="10.244.0.1:55426" http.request.uri="/v2/" http.request.useragent="Go 1.1 package http"     http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.089664ms http.response.status=200 http.response.written=2 instance.id=014d0ce0-acc1-4332-8352-    ab8f173ec830 service=registry version=v2.1.1-495-g1a90b2b 
2016-04-12T08:05:52.744024440Z 10.244.0.1 - - [12/Apr/2016:08:05:52 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go 1.1 package http"
2016-04-12T08:05:52.747036794Z time="2016-04-12T08:05:52.746943294Z" level=info msg="response completed" go.version=go1.5.3 http.request.host="10.244.0.6:5000" http.request.    id=f07dba93-ee92-4d9b-b825-42ecd9b08a06 http.request.method=GET http.request.remoteaddr="10.244.0.1:55425" http.request.uri="/v2/" http.request.useragent="Go 1.1 package http"     http.response.contenttype="application/json; charset=utf-8" http.response.duration=75.787271ms http.response.status=200 http.response.written=2 instance.id=014d0ce0-acc1-4332-8352    -ab8f173ec830 service=registry version=v2.1.1-495-g1a90b2b 
2016-04-12T08:05:52.764476772Z 10.244.0.1 - - [12/Apr/2016:08:05:52 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go 1.1 package http"
...
bacongobbler commented

The registry service listens on port 80, but the registry itself listens on port 5000. It looks like you missed a step because the builder is trying to authenticate via https: http://docs-v2.readthedocs.org/en/latest/installing-workflow/system-requirements/#docker-insecure-registry

arkkanoid commented

Hi! It works setting an inscure registry: https://coreos.com/os/docs/latest/registry-authentication.html.
Now, when I run Deis with this new cluster configuration I've an high CPU usage of Registry component:
On AWS EC3 r3.large:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12874 root 20 0 5268 1912 1508 R 93.4 0.0 1942:57 registry

image

I tested with S3 and minio object storage with the same result.
I'm running Deis v2-beta1

rimusz commented

@arkkanoid update to Deis v2-beta2 as it has that high CPU registry fix

arkkanoid commented

thanks @rimusz