distribution/distribution

Private registry:2 push fail: unable to ping registry endpoint...x509: cannot validate certificate for ... because it doesn't contain any IP SANs

BradleyA opened this issue ยท 16 comments

Unable to get TLS to work with private registry:2 on ubuntu 14.04
Private registry:2 push fail: unable to ping registry endpoint...x509: cannot validate certificate for ... because it doesn't contain any IP SANs

Private registry worked when I run it unsecure, but that is no help.

Unable to determine what is wrong with cert and/or docker registry:2

Created cert
...
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Texas
Locality Name (eg, city) []:Cedar Park
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Self
Organizational Unit Name (eg, section) []:IT
Common Name (e.g. server FQDN or YOUR name) []:192.168.1.102:5000
Email Address []:xxx.yyy@ymail.com
...
sudo -i
mkdir -p /etc/docker/certs.d/192.168.1.102:5000
chmod -R 700 /etc/docker/certs.d/192.168.1.102:5000
cd /mnt/three/TLS-cert/certs/
cp domain.crt /etc/docker/certs.d/192.168.1.102:5000/ca.crt
cd /etc/docker/certs.d/192.168.1.102:5000
ls -l /etc/docker/certs.d/192.168.1.102:5000
...
started docker -d -D
started docker run -d -p 5000:5000 ... registry:2

/mnt/three/TLS-cert/certs$ docker push 192.168.1.102:5000/mongo
The push refers to a repository [192.168.1.102:5000/mongo](len: 1)
unable to ping registry endpoint https://192.168.1.102:5000/v0/
v2 ping attempt failed with error: Get https://192.168.1.102:5000/v2/: x509: cannot validate certificate for 192.168.1.102 because it doesn't contain any IP SANs
v1 ping attempt failed with error: Get https://192.168.1.102:5000/v1/_ping: x509: cannot validate certificate for 192.168.1.102 because it doesn't contain any IP SANs

?

~/mnt-three/docker-registry$ docker version
Client:
Version: 1.8.1
API version: 1.20
Go version: go1.4.2
Git commit: d12ea79
Built: Thu Aug 13 02:35:49 UTC 2015
OS/Arch: linux/amd64

Server:
Version: 1.8.1
API version: 1.20
Go version: go1.4.2
Git commit: d12ea79
Built: Thu Aug 13 02:35:49 UTC 2015
OS/Arch: linux/amd64

~/mnt-three/docker-registry$ docker info
Containers: 3
Images: 332
Storage Driver: aufs
Root Dir: /mnt/three/docker-images/aufs
Backing Filesystem: extfs
Dirs: 350
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.16.0-46-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 8
Total Memory: 11.72 GiB
Name: three
ID: KIKV:2ITY:O5GF:5JEA:O5F6:GKKS:DYGG:NUMO:UTYK:B6ZR:OKB7:BECJ
Debug mode (server): true
File Descriptors: 21
Goroutines: 35
System Time: 2015-09-01T19:26:23.578926523-05:00
EventsListeners: 0
Init SHA1: f03be8ef95c082e1e68bfb533064420fcfd161eb
Init Path: /usr/lib/docker/dockerinit
Docker Root Dir: /mnt/three/docker-images
Username: allup2u72
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

/mnt/three/TLS-cert/certs$ docker exec 4703d2735d89 registry -version
registry github.com/docker/distribution v2.1.1

Command to launch registry
docker run -d -p 5000:5000 --name privte_registry
-v /mnt/three/docker-registry/registry-data:/tmp
-v /mnt/three/TLS-cert/certs:/certs
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt
-e REGISTRY_HTTP_TLS_KEY=/certs/domain.key
-e LOGLEVEL=INFO -e DEBUG=true
registry:2

docker daemon debug showing error:
...
DEBU[1211] Calling POST /images/{name:.*}/push
INFO[1211] POST /v1.20/images/192.168.1.102:5000/mongo/push?tag=
DEBU[1211] hostDir: /etc/docker/certs.d/192.168.1.102:5000
DEBU[1211] crt: /etc/docker/certs.d/192.168.1.102:5000/ca.crt
DEBU[1211] Trying to push 192.168.1.102:5000/mongo to https://192.168.1.102:5000 v2
DEBU[1211] Error getting v2 registry: Get https://192.168.1.102:5000/v2/: x509: cannot validate certificate for 192.168.1.102 because it doesn't contain any IP SANs
DEBU[1211] Trying to push 192.168.1.102:5000/mongo to https://192.168.1.102:5000 v1
DEBU[1211] hostDir: /etc/docker/certs.d/192.168.1.102:5000
DEBU[1211] crt: /etc/docker/certs.d/192.168.1.102:5000/ca.crt
DEBU[1211] attempting v2 ping for registry endpoint https://192.168.1.102:5000/v2/
DEBU[1211] attempting v1 ping for registry endpoint https://192.168.1.102:5000/v1/

registry log
...
2015/09/02 03:40:11 http: TLS handshake error from 172.17.42.1:57674: EOF
2015/09/02 03:45:03 http: TLS handshake error from 192.168.1.102:53321: remote error: bad certificate
2015/09/02 03:45:03 http: TLS handshake error from 192.168.1.102:53322: remote error: bad certificate
2015/09/02 03:45:03 http: TLS handshake error from 192.168.1.102:53323: remote error: bad certificate

command to create self signed certs
openssl req
-newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key
-x509 -days 365 -out certs/domain.crt

dmp42 commented

You need to use a domain name, or have the ip into your certificate.

More about that topic here (this is not a registry specific topic): http://serverfault.com/questions/611120/failed-tls-handshake-does-not-contain-any-ip-sans

Thank you for this direction to the correct information.

I had reviewed this page several times with many many other web posts about TLS and docker registry error messages. I did not understand that this logstash solution and docker registry:2 solution with TLS was the same incident, thank you.

I am running a proof of concept with docker without outside help of a security team. This is being setup on a group of four servers that are isolated without DNS to determine what business process changes may be needed for a move to a secure docker.

  1. Stopped and removed the running docker registry:2

  2. Edited the file /etc/ssl/openssl.cnf on the registry:2 host and added
    subjectAltName = IP:192.168.2.102 into the [v3_ca] section. Like the following:

    โ€ฆ
    [ v3_ca ]
    subjectAltName = IP:192.168.1.102
    ...

  3. Recreated the certificate using the same steps and information as defined above

  4. Copied the new certificate using the same steps as defined above on all four hosts

  5. Started registry:2 image using the same steps as defined above

  6. Tested docker push to registry:2 from two hosts and it works.

    /mnt-three/TLS-cert$ docker push 192.168.1.102:5000/python
    The push refers to a repository [192.168.1.102:5000/python](len: 1)
    e1857ee1f3b5: Image successfully pushed
    ...
    902b87aaaec9: Image successfully pushed
    2.7: digest: sha256:6da1183aeae37865eadc65cf0d93d68d1d766104bc8c8f32bf772eb87b5a87e0 size: 25093

Hopefully this information will be helpful to others and save them many web search hours.

dmp42 commented

@BradleyA thanks for this.
Happy you have it running!

It is very helpful. Thank U.

@BradleyA, thanks for the information. It worked for me.

Thanks, very helpful.

Thanks, @BradleyA!

cgebe commented

@BradleyA Thank you very much. Wasted a lot of time. Forgot to set the IP: prefix in subjectAltName.

@BradleyA thank you very very much!

Thanks a lot @BradleyA !
In my case I'm using CentOS so the config file was located in /etc/pki/tls/openssl.cnf

@BradleyA Thanks a lot for sharing this! and @djuretic thanks for pointing to the config for CentOS.
This works for me!

Thanks @BradleyA worked for me too

Thanks @BradleyA ,you save my life .

To create a certificate along with subjectAltName

openssl req \
-x509 \
-newkey rsa:4096 \
-sha256 \
-days 3560 \
-nodes \
-keyout certs/domain.key \
-out certs/domain.crt \
-subj '/CN=myregistrydomain.com' \
-extensions san \
-config <( \
  echo '[req]'; \
  echo 'distinguished_name=req'; \
  echo '[san]'; \
  echo 'subjectAltName=IP:127.0.0.1')

Thank for the openssl command @shrhawk-entertainer It worked for me!

SO Cento7 the file is:

/etc/pki/tls/openssl.cnf

and put this configuration on the hosts client where I pull the images

{ "insecure-registries" : ["myregistry.com:5000"] }

in the file /etc/docker/daemon.json