Cannot show history or delete image (S3 storage)

Question

Cannot show history or delete image (S3 storage)

mzylowski opened this issue 5 years ago · 10 comments

Hello!
I decided to start a new issue because I don't have any idea what else I can try to make this work.

I deployed secure docker registry (with certs and auth) and this registry is fully functional (I can push and pull images from other nodes after docker login). Registry is behind nginx-proxy but this doesn't change anything - my issue is also reproducible without nginx-proxy by directly calling registry via 5000 port.

Here is my registry config file:

version: 0.1                                                                                                                                  [2/1010]
log:
  fields:
    service: registry
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  s3:
    accesskey: MINIO_USER
    secretkey: MINIO_SECRET
    region: us-east-1
    regionendpoint: MINIO_ADDRESS
    bucket: registry
    encrypt: false
    secure: false
    v4auth: true
    chunksize: 5242880
    rootdirectory: /
http:
  addr: :443
  headers:
    X-Content-Type-Options: [nosniff]
    Access-Control-Allow-Origin: ['http://registry.zylowski.net:8000']
    Access-Control-Allow-Methods: ['HEAD', 'GET', 'OPTIONS', 'DELETE']
    Access-Control-Allow-Headers: ['Authorization']
    Access-Control-Max-Age: [1728000]
    Access-Control-Allow-Credentials: [true]
    Access-Control-Expose-Headers: ['Docker-Content-Digest']
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
auth:
  htpasswd:
    realm: basic-realm
    path: /auth/htpasswd

This how to I start my docker registry (docker run equivalent):

                       docker run \
                        -e "REGISTRY_AUTH=htpasswd" \
                        -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
                        -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
                        -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
                        -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
                        --net=zylowski.net --ip=172.18.0.102 \
                        --name=docker-registry \
                        -v /opt/registry/config.yml:/etc/docker/registry/config.yml \
                        -v /opt/registry/certs:/certs \
                        -v /opt/registry/auth:/auth \
                        registry:2

And the UI starts via:

docker run -d -p 8000:80 -e URL=https://registry.zylowski.net:443 -e DELETE_IMAGES=true joxit/docker-registry-ui:latest

Loging in browser works well:

Also after logging list of repositories are available:

But clicking on
causes infinity loading for images,
click on
causes

I believe Access-Control fields from config are set properly, especially when list of repositories and auth works well.

Here is the log from registry container, after click in thrash bin icon:

kwi 06 13:53:15 k8s-storage docker[24033]: time="2019-04-06T13:53:15.822591136Z" level=warning msg="error authorizing context: basic authentication challenge for realm "Registry Realm": invalid authorization credential" go.version=go1.11.2 http.request.host=registry.zylowski.net http.request.id=93cc1313-db3e-4bcb-a24c-ac2cadec2ae0 http.request.method=HEAD http.request.referer="http://registry.zylowski.net:8000/" http.request.remoteaddr=194.99.105.228 http.request.uri="/v2/ubuntu/manifests/14.04" http.request.useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" vars.name=ubuntu vars.reference=14.04
kwi 06 13:53:15 k8s-storage docker[24033]: 172.18.0.100 - - [06/Apr/2019:13:53:15 +0000] "HEAD /v2/ubuntu/manifests/14.04 HTTP/1.0" 401 149 "http://registry.zylowski.net:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36"
kwi 06 13:53:16 k8s-storage docker[24033]: time="2019-04-06T13:53:16.092275204Z" level=info msg="authorized request" go.version=go1.11.2 http.request.host=registry.zylowski.net http.request.id=60589348-0eb0-4d2e-b7b0-54e8bb1a1853 http.request.method=HEAD http.request.referer="http://registry.zylowski.net:8000/" http.request.remoteaddr=194.99.105.228 http.request.uri="/v2/ubuntu/manifests/14.04" http.request.useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" vars.name=ubuntu vars.reference=14.04
kwi 06 13:53:16 k8s-storage docker[24033]: time="2019-04-06T13:53:16.096250901Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=registry.zylowski.net http.request.id=60589348-0eb0-4d2e-b7b0-54e8bb1a1853 http.request.method=HEAD http.request.referer="http://registry.zylowski.net:8000/" http.request.remoteaddr=194.99.105.228 http.request.uri="/v2/ubuntu/manifests/14.04" http.request.useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" http.response.duration=9.982045ms http.response.status=304 http.response.written=0
kwi 06 13:53:16 k8s-storage docker[24033]: 172.18.0.100 - - [06/Apr/2019:13:53:16 +0000] "HEAD /v2/ubuntu/manifests/14.04 HTTP/1.0" 304 0 "http://registry.zylowski.net:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36"
kwi 06 13:53:16 k8s-storage docker[24033]: time="2019-04-06T13:53:16.164573765Z" level=warning msg="error authorizing context: basic authentication challenge for realm "Registry Realm": invalid authorization credential" go.version=go1.11.2 http.request.host=registry.zylowski.net http.request.id=5190575e-a1e2-4a83-ad4e-4febcc171f99 http.request.method=OPTIONS http.request.referer="http://registry.zylowski.net:8000/" http.request.remoteaddr=194.99.105.228 http.request.uri="/v2/ubuntu/manifests/sha256:187b4f1d2adc16cc50149ff2aff054feb673e23566dad0132f5973a4c7fef1f0" http.request.useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" vars.name=ubuntu vars.reference="sha256:187b4f1d2adc16cc50149ff2aff054feb673e23566dad0132f5973a4c7fef1f0"
kwi 06 13:53:16 k8s-storage docker[24033]: 172.18.0.100 - - [06/Apr/2019:13:53:16 +0000] "OPTIONS /v2/ubuntu/manifests/sha256:187b4f1d2adc16cc50149ff2aff054feb673e23566dad0132f5973a4c7fef1f0 HTTP/1.0" 401 87 "http://registry.zylowski.net:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36"

Answer 1 · 2019-04-09T00:48:27.000Z

Hello,
Thanks for your issue.
I don't seen anything wrong in your configuration (excepted the docker-registry-ui tag which should be static and not latest but it's may be a typo?)

I will try to see deeply your issue next week, because I am in Japan now

Answer 2 · 2019-04-09T07:11:22.000Z

Yep I tried a lot of things and I copied here wrong line. I use static tag.

Answer 3 · 2019-04-10T14:29:53.000Z

Hello,
I can not see the Trash can icon available. ( I would like to delete an image)
Here are my registry config and compose file:

version: 0.1
log:
  fields:
    service: registry
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
    Access-Control-Allow-Origin: ['http://localhost']
    Access-Control-Allow-Methods: ['HEAD', 'GET', 'OPTIONS', 'DELETE']
    Access-Control-Allow-Headers: ['Authorization']
    Access-Control-Max-Age: [1728000]
    Access-Control-Allow-Credentials: [true]
    Access-Control-Expose-Headers: ['Docker-Content-Digest']
auth:
  htpasswd:
    realm: basic-realm
    path: /etc/docker/registry/htpasswd

version: '2.0'
services:
  registry:
    image: registry:2.6.2
    restart: always
    ports:
      - 5000:5000
    environment:
      - REGISTRY_HTTP_HEADERS_Access-Control-Allow-Origin=['*']
    volumes:
      - ./registry-data:/var/lib/registry
      - ./registry-config/credentials.yml:/etc/docker/registry/config.yml
      - ./registry-config/htpasswd:/etc/docker/registry/htpasswd
    networks:
      - registry-ui-net

  ui:
    image: joxit/docker-registry-ui:static
    restart: always
    ports:
      - 80:80
    environment:
      - REGISTRY_TITLE=My Private Docker Registry
      - REGISTRY_URL=http://registry:5000
    depends_on:
      - registry
    networks:
      - registry-ui-net

networks:
  registry-ui-net:

Any ideas?

Answer 4 · 2019-04-10T22:38:22.000Z

Hi @RANGERBEE, for the delete icon you need this environment on the ui DELETE_IMAGES=true

Answer 5 · 2019-04-11T10:23:58.000Z

Hi @Joxit. Thanks for App. Unfortunately i have the same issue with history and delete. and i also use AWS S3

Answer 6 · 2019-04-15T18:37:45.000Z

Hello,
I'm back at home and I may have some leads.

When you use s3 storage, docker private registry send 307 status code (redirect) to your bucket.
That means your bucket should also be well configured (DELETE, Access-Control-Allow-Origin...)

Can you open a console and tell me what you have ?

I wrote an example here https://github.com/Joxit/docker-registry-ui/tree/master/examples/issue-75 (without credentials in docker registry) and it seems to works.
When I add docker registry credentials, it fails because of Access-Control-Allow-Origin

Answer 7 · 2019-04-18T13:01:44.000Z

Hello!
I made few checks today.
Your example for issue-75 (without auth) works well after deployment via compose-up.
Also I checked my configuration by removing auth section from configs. Everything worked fine. Unfortunately is important for me to use auth with my registry.

Can you open a console and tell me what you have ?

What exactly additional info I can get for you?

I also found 307 http code (in docker logs of the registry container):

time="2019-04-18T12:14:02.821608537Z" level=info msg="response completed" go.version=go1.11.2 http.request.host=registry.zylowski.net http.request.id=6b31cf3c-d997-49b4-a56c-a93cbddcefc2 http.request.method=GET http.request.referer="http://registry.zylowski.net:8000/" http.request.remoteaddr=194.99.105.228 http.request.uri="/v2/github-updater/blobs/sha256:c0059e2cde4f63fbd5a2d82bf61daa6d70d2235c102aa07aba6c3b1146661da0" http.request.useragent="Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" http.response.contenttype="application/octet-stream" http.response.duration=11.267582ms http.response.status=307 http.response.written=0
172.18.0.100 - - [18/Apr/2019:12:14:02 +0000] "GET /v2/github-updater/blobs/sha256:c0059e2cde4f63fbd5a2d82bf61daa6d70d2235c102aa07aba6c3b1146661da0 HTTP/1.0" 307 0 "http://registry.zylowski.net:8000/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"

In my minio deployment logs I see only:

2019-04-18 12:34:57.542135 I | http: TLS handshake error from 194.99.105.228:29210: EOF

Maybe this one is a bit related:
minio/minio#4044

Answer 8 · 2019-04-18T17:44:58.000Z

The issue with minio and credentials are CORS.
When you are using some credentials with browsers, the remote server which need your credentials must send the respond request with these headers :

Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: domain.example.com

But Minio is sending Access-Control-Allow-Origin: * instead of the current domain. Because of this, your browser block all these requests. The wildcard can be used only when no credentials are needed.

Here is the type of error in your browser:

Access to XMLHttpRequest at 'http://127.0.0.1:9000/registry/docker/registry/v2/blobs/sha256/73/7328f6f8b41890597575cbaadc884e7386ae0acc53b747401ebce5cf0d624560/data...' (redirected from 'http://127.0.0.1:5000/v2/alpine/blobs/sha256:7328f6f8b41890597575cbaadc884e7386ae0acc53b747401ebce5cf0d624560') from origin 'null' has been blocked by CORS policy: The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'. The credentials mode of requests initiated by the XMLHttpRequest is controlled by the withCredentials attribute.

If you want to avoid this issue, you will need a proxy in front of your Minio with a correct Access-Control-Allow-Origin.

Answer 9 · 2019-04-19T13:13:56.000Z

Okey, now I understand a lot more.
First issue (showing history of a tag) I fixed by adding

  redirect:
    disable: true

to my registry config.yml (to storage section).

I will try to solve CORS issue related with deleting images in next weeks (by changing nginx-proxy configuration). As workaround is possible to run chrome with

--disable-web-security

Answer 10 · 2021-08-17T12:55:43.000Z

Hi, I have the same problem: it works well, except for showing the history and for deleting images. This is my docker-compose.yml, my server is called testserver.local in my private network at home (it's a Raspberry Pi) and the error that I see in the log of docker-registry-ui, when requesting the history or a delete, is msg="error authorizing context: basic authentication challenge for realm "Registry Realm": invalid authorization credential". The certificates of docker-registry are self-signed and I don't use HTTPS for docker-registry-ui yet. The browser that I use for docker-registry-ui asks for credentials and these seem to work.

Can you help?

version: '2.0'
services:
  registry:
    restart: unless-stopped
    image: registry:2.7.1
    container_name: docker-registry
    ports:
      - 5000:5000
    environment:
      REGISTRY_STORAGE_DELETE_ENABLED: 'true'
      REGISTRY_HTTP_ADDR: 0.0.0.0:5000
      REGISTRY_HTTP_TLS_CERTIFICATE: /certs/testserver.local.crt
      REGISTRY_HTTP_TLS_KEY: /certs/testserver.local.key
      REGISTRY_AUTH: htpasswd
      REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
      REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
      REGISTRY_HTTP_HEADERS_Access-Control-Allow-Origin: '[http://testserver.local:5001]'
      REGISTRY_HTTP_HEADERS_Access-Control-Allow-Credentials: '[true]'
      REGISTRY_HTTP_HEADERS_Access-Control-Allow-Methods: '[HEAD, GET, OPTIONS, DELETE]'
      REGISTRY_HTTP_HEADERS_Access-Control-Allow-Headers: '[Authorization, Accept]'
      REGISTRY_HTTP_HEADERS_Access-Control-Max-Age: '[1728000]'
      REGISTRY_HTTP_HEADERS_Access-Control-Expose-Headers: '[Docker-Content-Digest]'
    volumes:
      - /storage/registry:/var/lib/registry
      - /home/pi/Docker/docker-registry/certs:/certs
      - /home/pi/Docker/docker-registry/auth:/auth
    networks:
      - registry-ui

  ui:
    restart: unless-stopped
    image: joxit/docker-registry-ui:latest
    container_name: docker-registry-ui
    ports:
      - 5001:80
    environment:
      - REGISTRY_TITLE=My Private Docker Registry
      - REGISTRY_URL=https://testserver.local:5000
      - DELETE_IMAGES=true
      - SINGLE_REGISTRY=true
    depends_on:
      - registry
    networks:
      - registry-ui

networks:
  registry-ui: