Cannot push cf app with docker image due to invalid layer size, but everything works with docker daemon.
FloThinksPi opened this issue · 9 comments
Hi all,
we are experiencing strange effects when trying to push an app with a image from a specific registry. This is a follow up of the slack thread: https://cloudfoundry.slack.com/archives/C033RE5D6/p1558969411005300
Description
When pushing an app like below:
cf push testapp --docker-image dockerhub.js-soft.com:1304/ui5:1.65.1
The push fails because grootfs raises a layer size mismatch error, see below.
However when pulling the exact same Image with the docker daemon, everything works as it should. Also when validating the v2 API of the Docker Registry with CURL manually showed, that layer size in manifest and downloaded layers match as well as the layer sha checksums.
When pulling(grootfs copy) the Image manually in grootfs(commit 285739d like in the release), it also works without problems. So we suspect the error to be one layer above grootfs, although the specific layer mismatch error is raised from grootfs.
Logging and/or test output
> cf push testapp --docker-image dockerhub.js-soft.com:1304/ui5:1.65.1
...
Updating app testapp...
Mapping routes...
Stopping app...
Waiting for app to start...
Start unsuccessful
TIP: use 'cf logs testapp --recent' for more information
FAILED
> cf logs testapp --recent
2019-06-05T07:00:20.03+0000 [CELL/0] ERR Cell a17988bf-c1b5-4f62-baf5-46cddca1f159 failed to create container for instance e69d2ba1-03bd-4b08-6b9e-c66f: running image plugin create: pulling the image: streaming blob `sha256:6c40cc604d8e4c121adcb6b0bfe8bb038815c350980090e74aa5a6423f8f82c0`: layer size is different from the value in the manifest
2019-06-05T07:00:20.03+0000 [CELL/0] ERR : exit status 1
When pulling with docker daemon
docker pull dockerhub.js-soft.com:1304/ui5:1.65.1
1.65.1: Pulling from ui5
6c40cc604d8e: Pull complete
30c6e03cb91a: Pull complete
105f03a3abdf: Pull complete
a492669e5fd8: Pull complete
4237ece2d3d8: Pull complete
a6f167a88fa7: Pull complete
fe86b80e713d: Pull complete
6958dff07fcb: Pull complete
8bc5e9aa0fd5: Pull complete
ae81a4647dda: Pull complete
13c641d49342: Pull complete
7e57089a32fa: Pull complete
Digest: sha256:4c9956b5a0e29c19b2f604a9505db123889879da115792304380422918d0c819
Status: Downloaded newer image for dockerhub.js-soft.com:1304/ui5:1.65.1
Manual Validation
> repo=ui5 && curl -H "Authorization: Bearer $(curl -sSL "https://dockerhub.js-soft.com:1304/v2/token?service=dockerhub.js-soft.com:1304/v2/token" | jq --raw-output .token)" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" "https://dockerhub.js-soft.com:1304/v2/${repo}/manifests/1.65.1"
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 11299,
"digest": "sha256:a6fbd955ced166675d813290d2ddad3f675bae39f86da433d15755f4a580941e"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 2754728,
"digest": "sha256:6c40cc604d8e4c121adcb6b0bfe8bb038815c350980090e74aa5a6423f8f82c0"
},
.............
]
}
> repo=ui5 && curl -H "Authorization: Bearer $(curl -sSL "https://dockerhub.js-soft.com:1304/v2/token?service=dockerhub.js-soft.com:1304/v2/token" | jq --raw-output .token)" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" "https://dockerhub.js-soft.com:1304/v2/${repo}/blobs/sha256:6c40cc604d8e4c121adcb6b0bfe8bb038815c350980090e74aa5a6423f8f82c0" -o layer.gz -L
> stat -f%z layer.gz
2754728
> sha256sum layer.gz
6c40cc604d8e4c121adcb6b0bfe8bb038815c350980090e74aa5a6423f8f82c0 layer.gz
Steps to reproduce
Pull above image, it is accessible without credentials.
Note that it is only reproduce-able with this exact docker repository.
We where not able to reproduce this on other repositories.
Also every image on the Repo shows this issue.
However Docker Daemon as reference implementation works, as well as grootfs copy and manually validation showed no issue with the repo, so we excluded the possibility of a corrupt or miss-configured docker repository.
- GRR v1.19.1
- Guardian d694f9a
- Grootfs 285739d
- Linux kernel version 4.15.0-1039-aws
- Go version 1.10.4
- Private Repo Sonatype Nexus 3.16.1-02
Best
Florian
We have created an issue in Pivotal Tracker to manage this:
https://www.pivotaltracker.com/story/show/166465168
The labels on this github issue will be updated when the story is started.
Hey @FloThinksPi ,
Is the docker registry still running? I get the following error on docker pull:
→ docker pull dockerhub.js-soft.com:1304/ui5:1.65.1
Error response from daemon: Get https://dockerhub.js-soft.com:1304/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
hi all,
sorry for the inconvenience, dockerhub.js-soft.com:1304 is online again.
best, v.
Thanks!
I was able to pull for a couple of hours, unfortunately I started getting authorisation errors:
→ docker pull dockerhub.js-soft.com:1304/ui5:1.65.1
Error response from daemon: Get https://dockerhub.js-soft.com:1304/v2/ui5/manifests/1.65.1: unauthorized: access to the requested resource is not authorized
Would it be possible to make the image accessible without credentials again?
urgh, sorry again. fat fingers and such...
anon pull is definitely on now
We found that in the case a Docker registry doesn't return Content-Length in the response headers (as is happening here), containers/image will set the size to -1. According to the Docker HTTP API docs, the request is not required to return Content-Length, so GrootFS should work around it in this case.
The fix will be available in the next release.
hm, @Callisto13's explanation makes sense. But in this case, Content-Length is returned:
$> repo=ui5 && curl -H "Authorization: Bearer $(curl -sSL "https://dockerhub.js-soft.com:1304/v2/token?service=dockerhub.js-soft.com:1304/v2/token" | jq --raw-output .token)" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" "https://dockerhub.js-soft.com:1304/v2/${repo}/manifests/1.65.1" -i
(...)
Content-Length: 2828
(...)
$> repo=ui5 && curl -H "Authorization: Bearer $(curl -sSL "https://dockerhub.js-soft.com:1304/v2/token?service=dockerhub.js-soft.com:1304/v2/token" | jq --raw-output .token)" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" "https://dockerhub.js-soft.com:1304/v2/${repo}/blobs/sha256:6c40cc604d8e4c121adcb6b0bfe8bb038815c350980090e74aa5a6423f8f82c0" -i
(...)
Content-Length: 2754728
(...)
Or am I seeing sth wrong here?
This is very weird, we logged the responses we were getting from the registry, and they did not have the Content-Length header:
Request URL: https://dockerhub.js-soft.com:1304/v2/ui5/blobs/sha256:6c40cc604d8e4c121adcb6b0bfe8bb038815c350980090e74aa5a6423f8f82c0
Headers: http.Header{
"Content-Security-Policy":[]string{"sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation"},
"Content-Type":[]string{"application/vnd.docker.image.rootfs.diff.tar.gzip"},
"Date":[]string{"Thu, 13 Jun 2019 09:25:38 GMT"},
"Docker-Content-Digest":[]string{"sha256:6c40cc604d8e4c121adcb6b0bfe8bb038815c350980090e74aa5a6423f8f82c0"},
"Docker-Distribution-Api-Version":[]string{"registry/2.0"},
"Last-Modified":[]string{"Fri, 05 Apr 2019 14:25:36 GMT"},
"Server":[]string{"Apache"},
"Vary":[]string{"Accept-Encoding"},
"X-Content-Type-Options":[]string{"nosniff"},
"X-Xss-Protection":[]string{"1; mode=block"}
}
So we tried the same with different HTTP clients: httpie (Python/requests) and Postman (Chrome/JS). None showed the Content-Length. It looks like curl is doing something magical 🤔
We are going to close this issue as it seems to be resolved.
Hello @gcapizzi @danail-branekov @Callisto13,
thank you for for the fast fix.
We've tested this version on 2 landscapes and it resolves this issue.
Also thanks for writing a regression test for this :).