HAproxy and registry push does not work together

[EugenMayer]

Current status:

• private registry:2 with nginx pass-proxy+ssl works ( using htauth ) sometimes work, see below
• private registry:2 with HA +ssl works ( using htauth ) does not work on push

We now know:

• portus like auth realms (token) do no have any influence, it neither works with htauth nor token auth with HA
• official signed certificates do not solve anything either

[kopax]

+1 at distribution/distribution#1754
Corrected by removing HAproxy

[EugenMayer]

created a test environment with the following docker compose file:

registry:
  container_name: registry
  image: registry:2
  volumes:
    - /data/registry_dw_test:/etc/docker/registry
nginx:
  container_name: registry
  image: "nginx:latest"
  ports:
    - 8443:443
  volumes:
    - /data/nginx_test/conf.d/domain.tld.conf:/etc/nginx/conf.d/domain.tld.conf
    - /data/ssl/domain.tld:/certs
  links:
    - registry:registry

with this nginx config

upstream docker-registry {
  server registry:5000;
}

server {
  listen 443;
  server_name domain.tld;

  # SSL
  ssl on;
  ssl_certificate /certs/production/certificates/domain.tld/fullchain.pem;
  ssl_certificate_key /certs/production/certificates/domain.tld/privkey.pem;

  # disable any limits to avoid HTTP 413 for large image uploads
  client_max_body_size 0;

  # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
  chunked_transfer_encoding on;

  location /v2/ {
    # Do not allow connections from docker 1.5 and earlier
    # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
    if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
      return 404;
    }

    # To add basic authentication to v2 use auth_basic setting plus add_header
    # auth_basic "registry.localhost";
    # auth_basic_user_file /etc/nginx/conf.d/registry.password;
    # add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

    proxy_pass                          http://docker-registry;
    proxy_set_header  Host              $http_host;   # required for docker client's sake
    proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
    proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_read_timeout                  900;
  }
}

and this registry config:

version: 0.1
log:
  level: info
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
  secret: somesecret
  host: domain.tld
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
auth:
  htpasswd:
    realm: basic-realm
    path: /etc/docker/registry/auth.htpasswd

This setup does not work out of any reason, when i expose the docker port to 8443.
Does not work means, that docker login works, but when i push, i get:

docker push domain.tld:8443/alpine                                                                                                           1 ↵
The push refers to a repository [domain.tld:8443/alpine]
8f01a53880b9: Pushing [=====================================>             ] 3.631 MB/4.797 MB

Interestingly, this setup works, when i run a barebone nginx and the registry as a container under jessie. I use exactly the same nginx config, just a other domain, same letsencrypt certs for the domain and exactly the same registry config, just a different "host".

So it could be due to the version:

Working ones:

• working: nginx/1.6.2 (jessie)
• registry github.com/docker/distribution v2.3.1

Not Working one:

• not working: nginx/1.11.0 ( nginx:latest from hub.github.io)
• registry github.com/docker/distribution v2.4.1 .. also tried v2.3.1 does not work.

Current state:

• it seem not to be related to the registry version. Using the same registry version 2.3.1 does not fix the issue
• i first tried to expose 8443:443 since 443 was used by a different service. But to ensure it was not the issue, i also mapped 443:443 in the not working example, so exactly the same port mapping as in the working example - and it did still not work

[EugenMayer]

finally i seem to find the cause for both, the nginx and the HA issue. Its not the proxy configuration, its not the registry version, its not the auth realm ( portus / htpasswd ) - its the docker client.

Changing to dockertoolbox and therfore to Docker version 1.10.3, build 20f81dd i could push to every single setup. With the current docker for mac beta13 i cannot push to any setup, not even my current production system, which used to work and was the cause i changed the strategy to look for the issue elsewhere then the servers.

So its the docker version on the pushing client, not anything else

[EugenMayer]

working with HA, HA->registry, HA->registry+portus and nginx+registry

[EugenMayer]

With the newest docker for mac relase beta13.1 i could push without issues, it uses
Docker version 1.11.1, build 5604cbe

[kopax]

I have tried with 1.11.1 on debian 8, same build, doesn't work with HA.

[EugenMayer]

@kopax can you try to downgrade your client and verify that it works with lower versions?

[kopax]

I have tried also with 1.11, didn't work. As soon as I get time, not sure if I will.

[EugenMayer]

@kopax well its working for me, so i do not really mind if you don't find time. No need to force it :)

[kopax]

@EugenMayer I know, this is not what I am saying. I have already tried before. But your conf is slightly different from mine. HAproxy wasn't doing SSL, nginx was.