Push hangs at first layer
mayorovp opened this issue · 3 comments
Description
I cannot push anything on private registry, like this:
root@gateway:/home/ubuntu# docker push registry.mcs.local/alpine
Using default tag: latest
The push refers to repository [registry.mcs.local/alpine]
5af4f8f59b76: Pushing [=======> ] 1.065MB/7.377MB
This push is hang forever without any progress.
Tcpdump shows ACK packets with zero window from registry to docker (docker wants to send more bytes on HTTP connection, but registry never reads his tcp socket).
Reproduce
- start registry with image distribution/distribution:edge using s3 from vk cloud
- configure docker to use insecure private registry
- try to push something to it
- see nothing on docker side, and infinite s3/ListParts requests on registry side in logs (>10 requests per second)
Expected behavior
Push is successfull. Or returned useful error message.
At least write some useful error message to logs!
registry version
registry:2.8.3, distribution/distribution:edge
Additional Info
My registry configuration:
version: 0.1
log:
level: debug
storage:
s3:
accesskey: [redacted]
secretkey: [redacted]
region: kz-ast
regionendpoint: https://hb.kz-ast.vkcs.cloud
bucket: registry-images
rootdirectory: /own
loglevel: debug
redirect:
disable: false
delete:
enabled: false
cache:
blobdescriptor: inmemory
maintenance:
uploadpurging:
enabled: true
age: 168h
interval: 24h
dryrun: false
readonly:
enabled: false
http:
addr: 0.0.0.0:80
headers:
X-Content-Type-Options: [nosniff]
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
Looks like storage redirect don't matter (i tried both configurations)
Docker version:
root@gateway:/home/ubuntu# docker version
Client:
Version: 24.0.5
API version: 1.43
Go version: go1.20.3
Git commit: 24.0.5-0ubuntu1~22.04.1
Built: Mon Aug 21 19:50:14 2023
OS/Arch: linux/amd64
Context: default
Server:
Engine:
Version: 24.0.5
API version: 1.43 (minimum version 1.12)
Go version: go1.20.3
Git commit: 24.0.5-0ubuntu1~22.04.1
Built: Mon Aug 21 19:50:14 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.2
GitCommit:
runc:
Version: 1.1.7-0ubuntu1~22.04.1
GitCommit:
docker-init:
Version: 0.19.0
GitCommit:
I've tested this on AWS S3 and this works like charm. My suspicion is this is related to #3940
My suspiction is incorrect NextPartNumberMarker in s3 reply, but i still want useful error message.
I haven't checked your logs and was responding from GH mobile app, hence my marking the comment I made as suspicion.
Feel free to open a PR with a fix. We welcome all contributions.