bug: 401 Unauthorized for private image registries while fetching base image metadata

[mucahitkantepe]

Are you use the envd server?

• Yes, I am using the envd server.
• No, I am not using the envd server.

Describe the bug

When using any private image registry, although local docker setup is authenticated, envd build command fails with 401 Unauthorized response. Tested with ECR and Artifactory

To Reproduce

have an envd file like below

# syntax=v1
def build():
    base(image="your.private.registry/your-private-image:your-tag", dev=False)

then run

envd build -f build.envd

Expected behavior

No response

The docker info output

Client:
 Version:    24.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     /Users/xxx.yyy@zzz.com/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /Users/xxx.yyy@zzz.com/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/xxx.yyy@zzz.com/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.19
    Path:     /Users/xxx.yyy@zzz.com/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.4
    Path:     /Users/xxx.yyy@zzz.com/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/xxx.yyy@zzz.com/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/xxx.yyy@zzz.com/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  v0.12.0
    Path:     /Users/xxx.yyy@zzz.com/.docker/cli-plugins/docker-scout

Server:
 Containers: 3
  Running: 1
  Paused: 0
  Stopped: 2
 Images: 3
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 6
 Total Memory: 7.667GiB
 Name: docker-desktop
 ID: d790af64-cddd-44f5-b654-6ac1607d607d
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

The envd version output

envd: v0.3.23
  BuildDate: 2023-05-25T08:23:47Z
  GitCommit: 8a175ed71d59859f57592ddbe73203c4dd8dd6c5
  GitTreeState: clean
  GitTag: v0.3.23
  GoVersion: go1.19.9
  Compiler: gc
  Platform: darwin/arm64

Additional context

This is introduced with #1148
The issue does not exist with version 0.2.4

[kemingy]

Can you access this private registry locally? This requires your auth file in $HOME/.docker/config.json.

[mucahitkantepe]

Yes, I can. This works when the base image is public but the push address is a private registry.

[mucahitkantepe]

I am not sure if it's related but only thing I can see is that buildkit v0.11.6 has a fix for some 401 Unauthorized issues. https://github.com/moby/buildkit/releases/tag/v0.11.6
moby/buildkit#3779 could we bump up the buildkit version?