docker-compose does not use stored credentials for private registry

[teutat3s]

It seems that docker-compose on sdc-docker does not read stored credentials of private registries after doing a docker login.
A pull-image-v2-1.0.0 of a new image from a private registry fails. Issuing a docker pull immediately afterwards works fine. So the stored credentials work fine for pure docker commands.

PI: 20200103T034110Z
docker0: docker release-20200102-20200103T002424Z-g3faf7f9

I would gladly provide more logs to debug this further if needed.

Command output:

$ docker-compose -f green-docker-compose.yml -p green up -d --scale vault=2
Pulling vault (hub.example.com/autopilotpattern-vault:1.3.2-TLS)...
ERROR: Error: image hub.example.com/autopilotpattern-vault:1.3.2-TLS not found (87f71b15-1847-48a7-ac03-a73ac7a9bbf4)

$ docker pull hub.example.com/autopilotpattern-vault:1.3.2-TLS
1.3.2-TLS: Pulling from hub.example.com/autopilotpattern-vault (req 243eb2e3-615a-4b15-a5c8-ba08f655f2ca)
c9b1b535fdd9: Already exists
9f72f270c653: Pull complete
4e08148aa39b: Pull complete
ec8428880d6e: Pull complete
17498c8b7a63: Pull complete
25a5cd892627: Pull complete
4e2f21dd706f: Pull complete
25eb0383c597: Pull complete
290713e940bc: Pull complete
ad1d523398f8: Pull complete
a04054a0eb36: Pull complete
334dad6a12c9: Pull complete
6389e31af7ba: Pull complete
5ca9e80e9fa3: Pull complete
Digest: sha256:64821ee9e8ac22184653c345619be426e1a36e1a31b5848f40809ba13d99fd1a
Status: Downloaded newer image for hub.example.com/autopilotpattern-vault:1.3.2-TLS
hub.example.com/autopilotpattern-vault:1.3.2-TLS

Logs of failed pull-image-v2-1.0.0 with docker-compose:

Task Summary
pull_image_v2_layers
UnauthorizedError

"execution": "failed",
  "chain_results": [
    {
      "result": "",
      "error": {
        "name": "ResourceNotFound",
        "message": "UnauthorizedError"
      },
      "name": "pull_image_v2_layers",
      "started_at": "2020-01-31T15:35:52.367Z",
      "finished_at": "2020-01-31T15:35:52.552Z"
    }

[twhiteman]

I'll take a look at this - but in the meantime could you provide the versions of:

• docker-compose
• docker
• sdc-docker

[teutat3s]

compose: 1.25.2
docker: 19.03.5
sdc-docker: docker release-20200102-20200103T002424Z-g3faf7f9

[twhiteman]

So far I cannot reproduce this - at least using a private registry from quay.io:

$ docker-compose-1.25 up -d
Pulling sleepy (quay.io/twhiteman/hello-sleeper:)...
latest: Pulling from quay.io/twhiteman/hello-sleeper (req e2eaf7e9-1152-4326-b7b9-2b83c388f12a)
a98bfe784296: Pull complete
Digest: sha256:a3756c155799930bf7e2febe755a0391b41c3fa9a5ce18794c1e3fa4d0955856
Status: Downloaded newer image for quay.io/twhiteman/hello-sleeper:latest
Creating quayio_sleepy_1 ... done

What version of the docker registry does your hub.example.com run?

[teutat3s]

I updated the registry to the latest version now - 2.7.1.
No change. Maybe the reason is I am using basic auth with an nginx reverse proxy as endpoint before the registry?
But docker login works fine... hmmm

EDIT:
I've also found a similar issue reported here, but should be resolved in compose 1.25 as per the changelog.
Also tried changing the "credsStore" in ~/.docker/config.json from desktop to osxkeychain - no difference.

If I change the docker endpoint from triton to my local docker engine, pulling with docker-compose works fine...

[twhiteman]

There does seem to be a lot of reports of docker compose with Basic Auth failing, I wonder if an older version of docker-compose (say 1.23) works with Triton?
https://github.com/docker/compose/releases/tag/1.23.2

[teutat3s]

Made no difference, here are verbose outputs of docker-compose and adminui in a gist

https://gist.github.com/teutat3s/c57e3032d33790c11afb36ee61611fe3

[twhiteman]

Thanks for testing that.

I've setup a basic auth registry in my local environment (using self-signed SSL certificates), which seems to work, and to my surprise also works correctly for triton as well.

Note that I'm using a custom hostname "myreg.triton" - so I had to (in the IMGAPI zone) add a /etc/hosts entry to map the hostname "myreg.triton" to the correct ip address.

Here's more details of what I ran and what I've configured:

$ export | grep DOCKER_HOST
declare -x DOCKER_HOST="tcp://dingo.triton:2376"

$ docker-compose pull
Pulling alp (myreg.triton/alpine:latest)...
/usr/local/Cellar/docker-compose/1.15.0/libexec/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for dingo.triton has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
latest: Pulling from myreg.triton/alpine (req a893cee0-de4a-46b0-8c1a-496c29879364)
c9b1b535fdd9: Pull complete
Digest: sha256:ddba4d27a7ffc3f86dd6c2f92041af252a1f23a8e742c90e6e1297bfa1bc0c45
Status: Downloaded newer image for myreg.triton/alpine:latest

$ docker --version
Docker version 17.06.0-ce, build 02c1d87
$ docker-compose --version
docker-compose version 1.15.0, build unknown

# sdcadm insts docker
INSTANCE                              SERVICE  HOSTNAME  VERSION                                     ALIAS
aec18af5-6c04-40cb-8287-df96505f5cfa  docker   headnode  release-20200130-20200129T191403Z-g3faf7f9  docker0

My registry is running version 2.6.2 in a ubuntu instance (real hardware):

/usr/bin/docker-registry --version
/usr/bin/docker-registry github.com/docker/distribution v2.6.2+debian

[teutat3s]

Somehow this started working for me, don't know which change in my setup caused it. Working with:

❯ docker-compose --version
docker-compose version 1.26.2, build unknown

❯ docker --version
Docker version 20.10.6, build v20.10.6

docker release-20210128-20210128T003947Z-ge75d796 (uuid: fe72a50a-339b-43cf-a086-fb4f4b655407)

❯ cat ~/.config/docker/config.json
{
	"auths": {
		"hub.great.domain": {
			"auth": "long_hash"
		}
	}
}