containers/buildah

Changing value of the ENV via ARG doesn't break the cache

skeggr opened this issue · 6 comments

skeggr commented

Issue Description

Hello!
Should podman have the same behavior as docker with caching of ENV? It seems like it differs now. Look at this example:

FROM docker.io/busybox
ARG VAR
ENV VAR=$VAR

If you build with podman and pass different values of VAR via --build-arg - it won't break build cache for line 2 in Containerfile (and it's OK), but it won't cause cache miss for line 3 as well. And it's quite strange. Is this expected behavior?

P.S. There was the issue about overriding ARG in ENV, but it's not my case

Steps to reproduce the issue

Steps to reproduce the issue

  1. Write simple Containerfile
FROM docker.io/busybox
ARG VAR
ENV VAR=$VAR
  1. Run
    podman build --build-arg VAR=1 -f Containerfile
  2. Run podman build again with changed VAR:
    podman build --build-arg VAR=2 -f Containerfile

Describe the results you received

First run:

# podman build --build-arg VAR=1 -f Containerfile
STEP 1/3: FROM docker.io/busybox
STEP 2/3: ARG VAR
--> dbe567ffbfcf
STEP 3/3: ENV VAR=$VAR
COMMIT
--> 1eae01e0c3f9
1eae01e0c3f9d8da8ed72dcd7528d1b5b2a7ede9999d0e646c4e01abb7234718

Second run, assigning new value to ENV (via ARG) doesn't break the build cache:

# podman build --build-arg VAR=2 -f Containerfile
STEP 1/3: FROM docker.io/busybox
STEP 2/3: ARG VAR
--> Using cache dbe567ffbfcfaa16a8c869c1c6d527ad9bb3d8c058a0ec1e63b2e5386e75a65d
--> dbe567ffbfcf
STEP 3/3: ENV VAR=$VAR
--> Using cache 1eae01e0c3f9d8da8ed72dcd7528d1b5b2a7ede9999d0e646c4e01abb7234718
--> 1eae01e0c3f9
1eae01e0c3f9d8da8ed72dcd7528d1b5b2a7ede9999d0e646c4e01abb7234718

Describe the results you expected

Assignment new value to environment variable should break the build cache like as docker build does:

docker build --progress=plain --build-arg VAR=1 -f Containerfile .       
#0 building with "default" instance using docker driver

containers/podman#1 [internal] load .dockerignore
containers/podman#1 transferring context: 2B done
containers/podman#1 DONE 0.0s

containers/podman#2 [internal] load build definition from Containerfile
containers/podman#2 transferring dockerfile: 84B done
containers/podman#2 DONE 0.0s

containers/podman#3 [internal] load metadata for docker.io/library/busybox:latest
containers/podman#3 DONE 1.8s

containers/podman#4 [1/1] FROM docker.io/library/busybox@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79
containers/podman#4 resolve docker.io/library/busybox@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79 0.0s done
containers/podman#4 sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79 2.29kB / 2.29kB done
containers/podman#4 sha256:023917ec6a886d0e8e15f28fb543515a5fcd8d938edb091e8147db4efed388ee 528B / 528B done
containers/podman#4 sha256:a416a98b71e224a31ee99cff8e16063554498227d2b696152a9c3e0aa65e5824 1.46kB / 1.46kB done
containers/podman#4 sha256:3f4d90098f5b5a6f6a76e9d217da85aa39b2081e30fa1f7d287138d6e7bf0ad7 0B / 2.22MB 0.1s
containers/podman#4 sha256:3f4d90098f5b5a6f6a76e9d217da85aa39b2081e30fa1f7d287138d6e7bf0ad7 1.05MB / 2.22MB 0.5s
containers/podman#4 sha256:3f4d90098f5b5a6f6a76e9d217da85aa39b2081e30fa1f7d287138d6e7bf0ad7 2.22MB / 2.22MB 0.6s done
containers/podman#4 extracting sha256:3f4d90098f5b5a6f6a76e9d217da85aa39b2081e30fa1f7d287138d6e7bf0ad7
containers/podman#4 extracting sha256:3f4d90098f5b5a6f6a76e9d217da85aa39b2081e30fa1f7d287138d6e7bf0ad7 0.1s done
containers/podman#4 DONE 0.7s

containers/podman#5 exporting to image
containers/podman#5 exporting layers done
containers/podman#5 writing image sha256:5ae224202fe0d19f6ea2feaaea6f7da9f6affbe8211e1854d234e119934bb25e done
containers/podman#5 DONE 0.0s

Second run (pay attention on the image sha):

 docker build --progress=plain --build-arg VAR=2 -f Containerfile .
#0 building with "default" instance using docker driver

containers/podman#1 [internal] load .dockerignore
containers/podman#1 transferring context: 2B done
containers/podman#1 DONE 0.0s

containers/podman#2 [internal] load build definition from Containerfile
containers/podman#2 transferring dockerfile: 84B done
containers/podman#2 DONE 0.0s

containers/podman#3 [internal] load metadata for docker.io/library/busybox:latest
containers/podman#3 DONE 0.8s

containers/podman#4 [1/1] FROM docker.io/library/busybox@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79
containers/podman#4 CACHED

containers/podman#5 exporting to image
containers/podman#5 exporting layers done
containers/podman#5 writing image sha256:47ab87abc7ed768bbf0cb6819dd228e26e967b889902fa367799a21d791aef00 done
containers/podman#5 DONE 0.0s

podman info output

host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 41.21
    systemPercent: 8.22
    userPercent: 50.57
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: container
    version: "38"
  eventLogger: file
  freeLocks: 2048
  hostname: 260f48f10afa
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.0-56-generic
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1412259840
  memTotal: 16576032768
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.9.2-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.9.2
      commit: 35274d346d2e9ffeacb22cc11590b0266a23d634
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231004.gf851084-1.fc38.x86_64
    version: |
      pasta 0^20231004.gf851084-1.fc38.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc38.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 892928
  swapTotal: 1023406080
  uptime: 1030h 20m 42.00s (Approximately 42.92 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/shared
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.12-1.fc38.x86_64
      Version: |-
        fusermount3 version: 3.14.1
        fuse-overlayfs: version 1.12
        FUSE library version 3.14.1
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 59915612160
  graphRootUsed: 18569003008
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.7.0
  Built: 1695839078
  BuiltTime: Wed Sep 27 18:24:38 2023
  GitCommit: ""
  GoVersion: go1.20.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.0

Podman in a container

Yes

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

rhatdan commented

podman build is goaled to match docker buildkit build.

flouthoc commented

I'll take this.

flouthoc commented

@skeggr I checked this against buildkit and afaics buildkit is behaving similar to buildah ( unlike legacy docker ). Here buildah will try to match behavior with buildkit as compared to docker

[fl@fedora test]$ sudo docker buildx build --build-arg VAR=1 --progress=plain -t test .
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 71B done
#1 DONE 0.1s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.1s

#3 [internal] load metadata for docker.io/library/busybox:latest
#3 DONE 1.2s

#4 [1/1] FROM docker.io/library/busybox@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79
#4 CACHED

#5 exporting to image
#5 exporting layers done
#5 writing image sha256:5ae224202fe0d19f6ea2feaaea6f7da9f6affbe8211e1854d234e119934bb25e done
#5 naming to docker.io/library/test done
#5 DONE 0.0s
[fl@fedora test]$ sudo docker buildx build --build-arg VAR=5 --progress=plain -t test .
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 31B done
#1 DONE 0.1s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.1s

#3 [internal] load metadata for docker.io/library/busybox:latest
#3 DONE 0.5s

#4 [1/1] FROM docker.io/library/busybox@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79
#4 CACHED

#5 exporting to image
#5 exporting layers done
#5 writing image sha256:b853e716c724d05adb59718f26d94b8ed6a9467641a2503cae6f40940bf29868 done
#5 naming to docker.io/library/test done
#5 DONE 0.0s
[fl@fedora test]$
rhatdan commented

Since we match docker with buildkit, I am closing this issue, feel free to continue conversation here.

skeggr commented

I checked this against buildkit and afaics buildkit is behaving similar to buildah ( unlike legacy docker ). Here buildah will try to match behavior with buildkit as compared to docker

[fl@fedora test]$ sudo docker buildx build --build-arg VAR=1 --progress=plain -t test .
. . .
#5 writing image sha256:5ae224202fe0d19f6ea2feaaea6f7da9f6affbe8211e1854d234e119934bb25e done

[fl@fedora test]$ sudo docker buildx build --build-arg VAR=5 --progress=plain -t test .
. . .
#5 writing image sha256:b853e716c724d05adb59718f26d94b8ed6a9467641a2503cae6f40940bf29868 done

Thanks for the answer. Well, in this docker buildx build log image SHAs are not the same. If you'll run containers from these images you can see that 'VAR' environment variable has different value inside the containers. It's OK and it is that I've tried to describe in the issue. And yes - buildah has the same (expected) behavior:

/opt # buildah build-using-dockerfile --build-arg VAR=1 .
STEP 1: FROM docker.io/busybox
Getting image source signatures
Copying blob 3f4d90098f5b done
Copying config a416a98b71 done
Writing manifest to image destination
Storing signatures
STEP 2: ARG VAR
STEP 3: ENV VAR=$VAR
STEP 4: COMMIT
Getting image source signatures
Copying blob 3d24ee258efc skipped: already exists
Copying blob 5f70bf18a086 done
Copying config f0f5165dae done
Writing manifest to image destination
Storing signatures
--> f0f5165daef6e22684e55a279583ffee806e916838f678ef9f339dfc5e4eb1b7

/opt # buildah build-using-dockerfile --build-arg VAR=2 .
STEP 1: FROM docker.io/busybox
STEP 2: ARG VAR
STEP 3: ENV VAR=$VAR
STEP 4: COMMIT
Getting image source signatures
Copying blob 3d24ee258efc skipped: already exists
Copying blob 5f70bf18a086 skipped: already exists
Copying config 6969da480f done
Writing manifest to image destination
Storing signatures
--> 6969da480f2bd98fd11ffa07392bfce9602aa6cab1eb2dbe833745bdc63cff47

There are two images as a result if you build with buildah.
But what about podman? It doesn't create another image if you pass new VAR value:

podman build --build-arg VAR=1 -f Dockerfile .
STEP 1/3: FROM docker.io/busybox
Trying to pull docker.io/library/busybox:latest...
Getting image source signatures
Copying blob 3f4d90098f5b done
Copying config a416a98b71 done
Writing manifest to image destination
STEP 2/3: ARG VAR
--> 1cc3146ad6f5
STEP 3/3: ENV VAR=$VAR
COMMIT
--> bc7c1c0a6d6f
bc7c1c0a6d6f57d30d330b255bde2007eacad224272caaaf35e78b0f0bcf4a43

# podman build --build-arg VAR=2 -f Dockerfile .
STEP 1/3: FROM docker.io/busybox
STEP 2/3: ARG VAR
--> Using cache 1cc3146ad6f524442e22d4dcb20c9932bbf3d9ea669789eb2f2bdb1fc869b288
--> 1cc3146ad6f5
STEP 3/3: ENV VAR=$VAR
--> Using cache bc7c1c0a6d6f57d30d330b255bde2007eacad224272caaaf35e78b0f0bcf4a43
--> bc7c1c0a6d6f
bc7c1c0a6d6f57d30d330b255bde2007eacad224272caaaf35e78b0f0bcf4a43