genuinetools/reg

clair.layer400 Bad Request: "vulnerability scanning for <image> failed: clair error: could not find layer"

tedsluis opened this issue · 2 comments

tedsluis commented

I am not able to get reg working. My setup is:

  • openshift 3.11
  • Openshift registry
  • clair 2.1.2 (running on openshift)
  • reg (running on openshift in a container based on redhat ubi image)

To access the openshift registry you need a token:

$MYTOKEN=$(oc whoami -t)

Using this token I can do for example a docker login:

$ docker login -u $USER -p $MYTOKEN <registry>

Now I am able to push and pull images.

I also use this token in reg and clair to read the manifest and image layers in the examples below:

Using reg form inside the reg container I can list images in the pls-clair inside my registry, using my openshift authentication token:

sh-4.2$ ./reg ls -u ted.sluis.ocp -p $MYTOKEN -d -k  docker-registry.default.svc:5000 list
INFO[0000] domain: docker-registry.default.svc:5000
INFO[0000] server address: docker-registry.default.svc:5000
pls-clair/clair                                                v2.1.2
pls-clair/reg                                                  v1.0
pls-clair/ubi                                                  latest
(left out other images)

I run reg sever in the reg container with the follow arguments:

$ reg server -d --clair http://clair:6060 -k -u ted.sluis -p $MYTOKEN  -r docker-registry.default.svc:5000 --asset-path /tmp --port 6006

It serves the static web page with the images in my registry and their individual tags. It is not able to service layer information.

This is what I see in the reg log when I try to view the pls-clair/clair:v2.1.2 image from the static web page:

time="2020-04-24T04:37:01Z" level=info msg="fetching vulnerabilities" URL="/repo/pls-clair%2Fclair/tag/v2.1.2/vulns" func=vulnerabilities method=GET
2020/04/24 04:37:01 registry.manifests uri=https://docker-registry.default.svc:5000/v2/pls-clair/clair/manifests/v2.1.2 repository=pls-clair/clair ref=v2.1.2
2020/04/24 04:37:01 registry.registry resp.Status=200 OK
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:8ef94372a977c02d425f12c8cbda5416e372b7a869a6c2b20342c589dba3eae5
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:8ef94372a977c02d425f12c8cbda5416e372b7a869a6c2b20342c589dba3eae5
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:1ec62c064901392a6722bb47a377c01a381f4482b1ce094b6d28682b6b6279fd
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:1ec62c064901392a6722bb47a377c01a381f4482b1ce094b6d28682b6b6279fd
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:339073ee4259ec00139ce0f376829e2c265f67aabe406e82cb6e2ee559ea1ea6
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:339073ee4259ec00139ce0f376829e2c265f67aabe406e82cb6e2ee559ea1ea6
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:df12825c8c18ed0861e6dda82b3c9cdca4c00b65d57e6d04f67d3e198fab3a06
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:df12825c8c18ed0861e6dda82b3c9cdca4c00b65d57e6d04f67d3e198fab3a06
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:a9915232c639134dda946eef2ca8dda4692be91be52b12a7ffe9d53d0ec0bf81
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:a9915232c639134dda946eef2ca8dda4692be91be52b12a7ffe9d53d0ec0bf81
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:91c388d3f8dfb80ca43c81d0c424483c7c3238ad175da1a68d4bae2d44e7a238
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:91c388d3f8dfb80ca43c81d0c424483c7c3238ad175da1a68d4bae2d44e7a238
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:4a33db4f96e98ba0b227eb9476f8931f07e8c38bc9793bcec65d90000bb8e855
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:4a33db4f96e98ba0b227eb9476f8931f07e8c38bc9793bcec65d90000bb8e855
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:97ac75448aa2cb011366f12171e36234a3418e4beccf911881dfe3dfdeb37a50
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:97ac75448aa2cb011366f12171e36234a3418e4beccf911881dfe3dfdeb37a50
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:901faf5e6502c7c8de28af6c73f08053ce2f69aeed3539c1612eb63acaaf5fd0
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:901faf5e6502c7c8de28af6c73f08053ce2f69aeed3539c1612eb63acaaf5fd0
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:cddc5806bc93e98d13f39a7e7fd8dc13bf27f772f6f9b0d9c0251b962afc0448
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:cddc5806bc93e98d13f39a7e7fd8dc13bf27f772f6f9b0d9c0251b962afc0448
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:f2e593b86155abf6c8a2c50fd6e086e76b0e2a68c52b07a6218810c5ccbaa3cc
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:f2e593b86155abf6c8a2c50fd6e086e76b0e2a68c52b07a6218810c5ccbaa3cc
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:66851844362af7571ea55958abee16ab5b59b2f9d084bf42629569ad7537dd9b
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:66851844362af7571ea55958abee16ab5b59b2f9d084bf42629569ad7537dd9b
2020/04/24 04:37:01 clair.ancestry.post name=sha256:3cad95957f1fee23e262cdd5bb084abcd827b2aba78edec194a217c09f98224e
2020/04/24 04:37:01 registry.manifests uri=https://docker-registry.default.svc:5000/v2/pls-clair/clair/manifests/v2.1.2 repository=pls-clair/clair ref=v2.1.2
2020/04/24 04:37:01 registry.registry resp.Status=200 OK
2020/04/24 04:37:01 registry.token url=https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17
2020/04/24 04:37:01 got empty token for https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17
2020/04/24 04:37:01 clair.layers.post url=http://clair:6060/v1/layers name=sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17
2020/04/24 04:37:01 clair.layers.post req.Body={"Layer":{"Name":"sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17","Path":"https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17","Format":"Docker"}}
2020/04/24 04:37:01 clair.layers.post resp.Status=400 Bad Request
time="2020-04-24T04:37:01Z" level=error msg="vulnerability scanning for pls-clair/clair:v2.1.2 failed: clair error: could not find layer" URL="/repo/pls-clair%2Fclair/tag/v2.1.2/vulns" func=vulnerabilities method=GET

And in the clair log I see this at the same time:

{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2020-04-24 04:37:00.296345","engine version":3,"format":"Docker","layer":"sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17","parent layer":"","path":"https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17"}
{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:136","Time":"2020-04-24 04:37:00.305892","status code":401}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2020-04-24 04:37:00.305950","error":"could not find layer","layer":"sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17","path":"https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2020-04-24 04:37:00.306556","elapsed time":10312151,"method":"POST","remote addr":"10.131.0.1:42472","request uri":"/v1/layers","status":"400"}
{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2020-04-24 04:37:01.847010","engine version":3,"format":"Docker","layer":"sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17","parent layer":"","path":"https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17"}
{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:136","Time":"2020-04-24 04:37:01.857130","status code":401}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2020-04-24 04:37:01.857229","error":"could not find layer","layer":"sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17","path":"https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2020-04-24 04:37:01.858427","elapsed time":11477492,"method":"POST","remote addr":"10.131.0.1:42472","request uri":"/v1/layers","status":"400"}

In the registry I see this at the same time:

time="2020-04-24T04:36:59.942638871Z" level=info msg="response completed" go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=db646aee-0caf-4f5d-b205-b6ed0a355796 http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/tags/list http.request.useragent=Go-http-client/1.1 http.response.contenttype="application/json; charset=utf-8" http.response.duration=9.320875ms http.response.status=200 http.response.written=45 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:36:59.942677086Z" level=info msg=response go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=bf37e1d9-4b36-4890-a683-65f5eccfb272 http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/tags/list http.request.useragent=Go-http-client/1.1 http.response.contenttype="application/json; charset=utf-8" http.response.duration=9.399815ms http.response.status=200 http.response.written=45 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:36:59.961723966Z" level=info msg="rewriting manifest sha256:55fb9b5af9a1862fea000da8157919f083de1cd328ce25cc1593f3321dc6ef3d in schema1 format to support old client" go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=01581626-3cab-48f3-be26-2217ddb10fc1 http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076 openshift.auth.user=Ted.Sluis.ocp openshift.auth.userid=28b463bc-020f-11ea-9541-005056b65da0 vars.name=pls-clair/clair vars.reference=v2.1.2
time="2020-04-24T04:36:59.969134594Z" level=info msg="response completed" go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=01581626-3cab-48f3-be26-2217ddb10fc1 http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v1+prettyjws http.response.duration=25.945578ms http.response.status=200 http.response.written=17097 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:36:59.969294048Z" level=info msg=response go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=9c430cdd-d6a1-4616-94b9-d260b031d33b http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v1+prettyjws http.response.duration=26.172736ms http.response.status=200 http.response.written=17097 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:00.083026722Z" level=info msg="response completed" go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=1af53c15-dd35-4fbb-9f5e-b5094ea62e5b http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=13.614697ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:00.083159483Z" level=info msg=response go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=1c4800cd-0383-46b5-9218-d7d8e5884d8e http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=13.788153ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:00.283528191Z" level=info msg="response completed" go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=38853b66-c344-405e-bfd7-9817f0b7b719 http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=11.819507ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:00.28356633Z" level=info msg=response go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=9737e4b1-c2bf-4d6a-a2ca-b42632442b23 http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=11.898217ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:01.640114926Z" level=info msg="response completed" go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=8eb8f2ab-6036-4091-9de3-138d95f51985 http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=26.200464ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:01.640267302Z" level=info msg=response go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=20beee50-2a8a-438e-8383-7c769aa3c4af http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=26.372037ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:01.834676388Z" level=info msg="response completed" go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=53ae0dc9-fcbb-4ede-8f8c-b0e2807a6f1a http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=12.667436ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076
time="2020-04-24T04:37:01.834711297Z" level=info msg=response go.version=go1.9.7 http.request.host="docker-registry.default.svc:5000" http.request.id=a2fe45ce-254b-41c3-9862-5d494d5964fe http.request.method=GET http.request.remoteaddr="10.131.0.1:60223" http.request.uri=/v2/pls-clair/clair/manifests/v2.1.2 http.request.useragent=Go-http-client/1.1 http.response.contenttype=application/vnd.docker.distribution.manifest.v2+json http.response.duration=12.738622ms http.response.status=200 http.response.written=3248 instance.id=d941f687-2b34-4ae0-a9e7-e76956233076

I am not sure what's going wrong. To troubleshoot I tried the following steps:

  1. Getting the manifest of an image from my registry using my token.
curl -k  -H "Authorization: Bearer $MYTOKEN" -X GET -i https://docker-registry.default.svc:5000/v2/pls-clair/clair/manifests/v2.1.2
HTTP/1.1 200 OK
Content-Length: 17097
Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
Docker-Content-Digest: sha256:a6d230227302affbd296d26b35293630af714c40ddd468d33ab0a92b9de25e74
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:a6d230227302affbd296d26b35293630af714c40ddd468d33ab0a92b9de25e74"
X-Registry-Supports-Signatures: 1
Date: Fri, 24 Apr 2020 06:27:10 GMT
{
   "schemaVersion": 1,
   "name": "pls-clair/clair",
   "tag": "v2.1.2",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
      },
      {
         "blobSum": "sha256:66851844362af7571ea55958abee16ab5b59b2f9d084bf42629569ad7537dd9b"

The full output can be found here: https://pastebin.com/FqCNP1fh

  1. Getting the blobs from the manifest of this image from my registry using my token.
$ curl -k  -H "Authorization: Bearer BZOCuMfx5psnc1OFurdJ6Nc_FHkJLQaqAcj5pZWHPVo" -X GET -i https://docker-registry.default.svc:5000/v2/pls-clair/clair/manifests/v2.1.2 | grep blob
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
         "blobSum": "sha256:66851844362af7571ea55958abee16ab5b59b2f9d084bf42629569ad7537dd9b"
         "blobSum": "sha256:f2e593b86155abf6c8a2c50fd6e086e76b0e2a68c52b07a6218810c5ccbaa3cc"
         "blobSum": "sha256:cddc5806bc93e98d13f39a7e7fd8dc13bf27f772f6f9b0d9c0251b962afc0448"
         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
1         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
0         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
0         "blobSum": "sha256:901faf5e6502c7c8de28af6c73f08053ce2f69aeed3539c1612eb63acaaf5fd0"
          "blobSum": "sha256:b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1"
1         "blobSum": "sha256:97ac75448aa2cb011366f12171e36234a3418e4beccf911881dfe3dfdeb37a50"
7         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
0         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
9         "blobSum": "sha256:4a33db4f96e98ba0b227eb9476f8931f07e8c38bc9793bcec65d90000bb8e855"
7         "blobSum": "sha256:91c388d3f8dfb80ca43c81d0c424483c7c3238ad175da1a68d4bae2d44e7a238"
          "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
          "blobSum": "sha256:a9915232c639134dda946eef2ca8dda4692be91be52b12a7ffe9d53d0ec0bf81"
1         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
0         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
0         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
          "blobSum": "sha256:df12825c8c18ed0861e6dda82b3c9cdca4c00b65d57e6d04f67d3e198fab3a06"
1         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
7         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
0         "blobSum": "sha256:339073ee4259ec00139ce0f376829e2c265f67aabe406e82cb6e2ee559ea1ea6"
9         "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
7         "blobSum": "sha256:1ec62c064901392a6722bb47a377c01a381f4482b1ce094b6d28682b6b6279fd"
          "blobSum": "sha256:8ef94372a977c02d425f12c8cbda5416e372b7a869a6c2b20342c589dba3eae5"
          "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
          "blobSum": "sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17"
  1. Downloading the blob from the registry using using my token:
sh-4.2$ curl -k  -H "Authorization: Bearer BZOCuMfx5psnc1OFurdJ6Nc_FHkJLQaqAcj5pZWHPVo" -X GET -i https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 > /tmp/layer

sh-4.2$ ls -l /tmp
-rw-r--r--. 1 1016010000 root 462 Apr 24 06:37 layer
  1. Post a layer to clair:
$ curl -k -X POST -i http://clair:6060/v1/layers -d '{"Layer": {"Name": "b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1", "Path": "https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1", "Headers": {  "Authorization": "Bearer BZOCuMfx5psnc1OFurdJ6Nc_FHkJLQaqAcj5pZWHPVo" }, "Format": "Docker", "ParentName": ""}}'
HTTP/1.1 201 Created
Content-Type: application/json;charset=utf-8
Server: clair
Date: Fri, 24 Apr 2020 06:54:24 GMT
Content-Length: 353

{"Layer":{"Name":"b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1","Path":"https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1","Headers":{"Authorization":"Bearer BZOCuMfx5psnc1OFurdJ6Nc_FHkJLQaqAcj5pZWHPVo"},"Format":"Docker","IndexedByVersion":3}}

The logging in the clair container:

{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2020-04-24 06:54:24.526477","engine version":3,"format":"Docker","layer":"b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1","parent layer":"","path":"https://docker-registry.default.svc:5000/v2/pls-clair/clair/blobs/sha256:b5f0c0e7dfb70f2b26036129f8af86ee5619868b8dc5a3d9191e6735fd020fc1"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2020-04-24 06:54:24.578878","elapsed time":52523668,"method":"POST","remote addr":"10.131.0.1:50514","request uri":"/v1/layers","status":"201"}

A http status 201 looks okay, I think (201 = result of HTTP POST request, one or more new resources have been successfully created on server).

Now back to reg and the error I got in the first place while running reg server:

2020/04/24 04:37:01 clair.layers.post resp.Status=400 Bad Request
time="2020-04-24T04:37:01Z" level=error msg="vulnerability scanning for pls-clair/clair:v2.1.2 failed: clair error: could not find layer" URL="/repo/pls-clair%2Fclair/tag/v2.1.2/vulns" func=vulnerabilities method=GET

According the clair v1 api documentation a http status 400 means "The body of the request invalid". Does reg request to clair in the correct way?

Were it I go wrong? How can I resolve this. or troubleshoot. Any feedback is welcome.

danielwindit commented

Hitting this same issue, same circumstances

danielwindit commented

Solved it in my case by adding out custom CA to the container running REG and pointing reg (-r) to the route of the internal registry of Openshift instead of the service.