When using extensions, if the last layer in the run image is duplicated, then the process fail with an error
Closed this issue · 4 comments
juancoen commented
Summary
When using extensions, if the run image latest layer is duplicated, then the process fails with an error like:
[extender (run)] ERROR: failed to extend run image: failed to copy extended image to output directory: open /kaniko/cache/base/sha256:f07ab0840dba11b097245ed458798aae52e6f5616f153e62825ca3ad4a7d3c69/blobs/sha256/b4af3bc866d42dd85a3862570db7d79172e169904d0def439abeae32a00dea16: no such file or directory
Reproduction
Steps
- Create a run image which latest layer is duplicated, for example:
FROM ...
RUN command
WORKDIR /usr/src/app
RUN another command
WORKDIR /usr/src/app- Use the image as a builder stack run-image
- Use an extension that adds something to the run-image
- Try to create an image using this builder
Current behavior
The build fails with this error:
[extender (run)] ERROR: failed to extend run image: failed to copy extended image to output directory: open /kaniko/cache/base/sha256:f07ab0840dba11b097245ed458798aae52e6f5616f153e62825ca3ad4a7d3c69/blobs/sha256/b4af3bc866d42dd85a3862570db7d79172e169904d0def439abeae32a00dea16: no such file or directory
Expected behavior
The build must complete successfully
Context
lifecycle version
0.18.4
platform version(s)
pack report
Pack:
Version: 0.32.1+git-b14250b.build-5241
OS/Arch: darwin/amd64
Default Lifecycle Version: 0.17.2
Supported Platform APIs: 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12
Config:
experimental = true
layout-repo-dir = "/Users/jcoen/.pack/layout-repo"
Client:
Version: 25.0.3
Context: desktop-linux
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.12.1-desktop.4
Path: /Users/jcoen/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.24.5-desktop.1
Path: /Users/jcoen/.docker/cli-plugins/docker-compose
debug: Get a shell into any image or container. (Docker Inc.)
Version: 0.0.24
Path: /Users/jcoen/.docker/cli-plugins/docker-debug
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.0
Path: /Users/jcoen/.docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.21
Path: /Users/jcoen/.docker/cli-plugins/docker-extension
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.4
Path: /Users/jcoen/.docker/cli-plugins/docker-feedback
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.0.0
Path: /Users/jcoen/.docker/cli-plugins/docker-init
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /Users/jcoen/.docker/cli-plugins/docker-sbom
scout: Docker Scout (Docker Inc.)
Version: v1.4.1
Path: /Users/jcoen/.docker/cli-plugins/docker-scout
WARNING: Plugin "/Users/jcoen/.docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /Users/jcoen/.docker/cli-plugins/docker-scan: no such file or directory
Server:
Containers: 16
Running: 2
Paused: 0
Stopped: 14
Images: 40
Server Version: 25.0.3
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
runc version: v1.1.12-0-g51d5e94
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.6.12-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 10
Total Memory: 62.69GiB
Name: docker-desktop
ID: 0e22bbb1-6f60-4587-b026-889ca9a4c6c5
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
anything else?
no
jericop commented
I'm not sure if this is related, but I'm getting a similar error when trying to follow the buildpacks.io guide to create an extension here.
Running on an amd64 mac.
pack version: 0.33.2+git-f2cffc4.build-5562
lifecycle version: 0.18.5
I also tried it with lifecycle versions 0.18.1, 0.18.2, 0.18.3, and 0.18.4. All failed with the same error.
➜ buildpacks pack --version
0.33.2+git-f2cffc4.build-5562
➜ buildpacks
➜ buildpacks docker run -d --rm -p 5000:5000 --name registry registry:2
9d9cc6bb0fafdd095d76d8ca9767ce8b9a4102cf502b36da6198ab6cf913a781
➜ buildpacks pack builder create localhost:5000/extensions-builder \
--config $PWD/samples/builders/alpine/builder.toml \
--publish
Successfully created builder image localhost:5000/extensions-builder
Tip: Run pack build <image-name> --builder localhost:5000/extensions-builder to use this builder
➜ buildpacks docker buildx build \
--file $PWD/samples/base-images/alpine/run/curl.Dockerfile \
--tag localhost:5000/run-image-curl --push .
[+] Building 0.7s (8/8) FINISHED docker:desktop-linux
=> [internal] load build definition from curl.Dockerfile 0.0s
=> => transferring dockerfile: 631B 0.0s
=> [internal] load metadata for docker.io/curlimages/curl:latest 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [1/3] FROM docker.io/curlimages/curl:latest 0.0s
=> CACHED [2/3] RUN curl --version 0.0s
=> CACHED [3/3] RUN addgroup -g 1001 cnb && adduser -u 1000 -G cnb -s /bin/bash -D cnb 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:d494e6196217dba06eed7432778d5ca892412d7da8cb7712738790c634d9fc93 0.0s
=> => naming to localhost:5000/run-image-curl 0.0s
=> pushing localhost:5000/run-image-curl with docker 0.5s
=> => pushing layer 3291863abd2f 0.1s
=> => pushing layer e2be6162e64d 0.1s
=> => pushing layer 8e406d5548c1 0.1s
=> => pushing layer b921b7bdd945 0.1s
=> => pushing layer 6094041ec9a7 0.1s
=> => pushing layer d4629ddd9445 0.1s
=> => pushing layer 606b0fac4a3a 0.2s
=> => pushing layer f5a6f6774bce 0.2s
=> => pushing layer f83640e95477 0.1s
=> => pushing layer 0ea665f430c2 0.1s
=> => pushing layer fe12ac6dc06c 0.3s
=> => pushing layer 221327e0f59c 0.4s
=> => pushing layer 994393dc58e7 0.5s
➜ buildpacks pack build hello-extensions \
--builder localhost:5000/extensions-builder \
--env BP_EXT_DEMO=1 \
--env BP_REQUIRES=vim,curl \
--path $PWD/samples/apps/java-maven \
--pull-policy always \
--network host \
--verbose
Builder localhost:5000/extensions-builder is untrusted
As a result, the phases of the lifecycle which require root access will be run in separate trusted ephemeral containers.
For more information, see https://medium.com/buildpacks/faster-more-secure-builds-with-pack-0-11-0-4d0c633ca619
Pulling image localhost:5000/extensions-builder:latest
latest: Pulling from extensions-builder
4abcf2066143: Already exists
156c2a1e46a7: Already exists
a86538552170: Already exists
1f0346ad010f: Already exists
f9c8ac9ab409: Already exists
f3fc0f991d83: Already exists
f10bc4ceffe5: Already exists
569fe741231d: Already exists
6d5077fd4ab0: Already exists
b7237553612e: Already exists
1c971575c8d6: Already exists
4da96f148d45: Already exists
90903455c218: Already exists
62b7c2336dfd: Already exists
9e90005d9e74: Already exists
c983951b913c: Already exists
4b2fe7e481b4: Already exists
32d2a27e557e: Already exists
25f51db0314e: Already exists
89732bc75041: Already exists
Digest: sha256:042ee2582ffa1fbddf5bf4e04bacb73084742e51216905ce7ebbe7c6371b948b
Status: Downloaded newer image for localhost:5000/extensions-builder:latest
CheckReadAccess succeeded for the run image cnbs/sample-base-run:alpine
Selected run image cnbs/sample-base-run:alpine
Pulling image cnbs/sample-base-run:alpine
alpine: Pulling from cnbs/sample-base-run
Digest: sha256:3d714eab0232f25bf14f3c5e8d15a4f813865728a5833d9cd9804d54928f871c
Status: Image is up to date for cnbs/sample-base-run:alpine
Pulling image buildpacksio/lifecycle:0.18.5
0.18.5: Pulling from buildpacksio/lifecycle
Digest: sha256:16a4c9e945381ad182fbb92710f0643e1c434ce11c3c43fb0ce6fcfce0d5efb7
Status: Image is up to date for buildpacksio/lifecycle:0.18.5
Creating builder with the following buildpacks:
-> samples/java-maven@0.0.2
-> samples/kotlin-gradle@0.0.2
-> samples/hello-universe@0.0.1
-> samples/hello-moon@0.0.1
-> samples/hello-world@0.0.1
-> samples/hello-extensions@0.0.1
Provided Environment Variables
BP_EXT_DEMO=1
BP_REQUIRES=vim,curl
Using build cache volume pack-cache-library_hello-extensions_latest-605c1c945967.build
===> ANALYZING
Running the analyzer on OS linux from image buildpacksio/lifecycle:0.18.5 with:
Container Settings:
Args: /cnb/lifecycle/analyzer -log-level debug -daemon -run /layers/run.toml -run-image cnbs/sample-base-run:alpine -launch-cache /launch-cache hello-extensions
System Envs: CNB_USER_ID=1000 CNB_GROUP_ID=1001 CNB_PLATFORM_API=0.12
Image: buildpacksio/lifecycle:0.18.5
User: root
Labels: map[author:pack]
Host Settings:
Binds: /var/run/docker.sock:/var/run/docker.sock pack-cache-library_hello-extensions_latest-605c1c945967.launch:/launch-cache pack-layers-sybjbqfvld:/layers pack-app-dygkglcczv:/workspace
Network Mode: host
[analyzer] Starting analyzer...
[analyzer] Parsing inputs...
[analyzer] Ensuring privileges...
[analyzer] Executing command...
[analyzer] Timer: Analyzer started at 2024-02-25T02:40:59Z
[analyzer] Found image with identifier "dbbad154b2ca1fe6de578774bb3ada003774a0fa55083ebfc6e5c29013124c5b"
[analyzer] Restoring data for SBOM from previous image
[analyzer] Retrieving previous image SBOM layer for "sha256:e45a9ce4189efaae8efbeb9ea4b634f225ded1a4b5711e7c0be6a2c97b457ad3"
[analyzer] Found image with identifier "006ca3744686da3362e56319682d0fd1f221dd9554a11424f1215e499b1555a6"
[analyzer] Timer: Analyzer ran for 1.029356ms and ended at 2024-02-25T02:40:59Z
[analyzer] Run image info in analyzed metadata is:
[analyzer] {"Reference":"006ca3744686da3362e56319682d0fd1f221dd9554a11424f1215e499b1555a6","Image":"cnbs/sample-base-run:alpine","Extend":false,"target":{"os":"linux","arch":"amd64"}}
===> DETECTING
Running the detector on OS linux from image pack.local/builder/6561706c71706363636f:latest with:
Container Settings:
Args: /cnb/lifecycle/detector -app /workspace -log-level debug
System Envs: CNB_EXPERIMENTAL_MODE=warn CNB_PLATFORM_API=0.12
Image: pack.local/builder/6561706c71706363636f:latest
User:
Labels: map[author:pack]
Host Settings:
Binds: pack-layers-sybjbqfvld:/layers pack-app-dygkglcczv:/workspace
Network Mode: host
[detector] Starting detector...
[detector] Parsing inputs...
[detector] Ensuring privileges...
[detector] Executing command...
[detector] Warning: Platform requested experimental feature 'Dockerfiles'
[detector] Timer: Detector started at 2024-02-25T02:40:59Z
[detector] Checking for match against descriptor: {linux []}
[detector] Checking for match against descriptor: {linux []}
[detector] Checking for match against descriptor: {linux []}
[detector] Checking for match against descriptor: {linux []}
[detector] ======== Results ========
[detector] pass: samples/vim@0.0.1
[detector] pass: samples/curl@0.0.1
[detector] pass: samples/cowsay@0.0.1
[detector] pass: samples/hello-extensions@0.0.1
[detector] Resolving plan... (try #1)
[detector] skip: samples/cowsay@0.0.1 provides unused cowsay
[detector] 3 of 4 buildpacks participating
[detector] samples/vim 0.0.1
[detector] samples/curl 0.0.1
[detector] samples/hello-extensions 0.0.1
[detector] Timer: Detector ran for 16.136781ms and ended at 2024-02-25T02:40:59Z
[detector] Timer: Generator started at 2024-02-25T02:40:59Z
[detector] Running generate for extension samples/vim@0.0.1
[detector] Looking up extension
[detector] Finding plan
[detector] Invoking command
[detector] Creating plan directory
[detector] Preparing paths
[detector] Running generate command
[detector] Reading output files
[detector] Found '1' Dockerfiles for processing
[detector] Finished running generate for extension samples/vim@0.0.1
[detector] Running generate for extension samples/curl@0.0.1
[detector] Looking up extension
[detector] Finding plan
[detector] Invoking command
[detector] Creating plan directory
[detector] Preparing paths
[detector] Running generate command
[detector] Reading output files
[detector] Found '1' Dockerfiles for processing
[detector] Finished running generate for extension samples/curl@0.0.1
[detector] Checking run image
[detector] Found a run.Dockerfile from extension 'samples/curl' setting run image to 'localhost:5000/run-image-curl'
[detector] Warning: new runtime base image 'localhost:5000/run-image-curl' not found in run metadata
[detector] Updating analyzed metadata with new run image 'localhost:5000/run-image-curl'
[detector] Copying Dockerfiles
[detector] Copying /tmp/cnb-extensions-generated.1096228511/samples_vim/build.Dockerfile to /layers/generated/build/samples_vim/Dockerfile
[detector] Copying /tmp/cnb-extensions-generated.1096228511/samples_curl/run.Dockerfile to /layers/generated/run/samples_curl/Dockerfile
[detector] Timer: Generator ran for 11.333892ms and ended at 2024-02-25T02:40:59Z
[detector] Run image info in analyzed metadata is:
[detector] {"Reference":"localhost:5000/run-image-curl","Image":"localhost:5000/run-image-curl","Extend":false}
Pulling image localhost:5000/run-image-curl
latest: Pulling from run-image-curl
Digest: sha256:9053673c6b256b37e95bfa1a5e65622a97d4a59187c8f6187410659f75d4b011
Status: Image is up to date for localhost:5000/run-image-curl:latest
ERROR: failed to build: executing lifecycle: open /var/folders/sd/l4j9q6l96bngknrk2xvdhjyh0000gq/T/extend-run-image-scratch778994993/blobs/sha256.tar: no such file or directory
➜ buildpacks
natalieparellano commented
Thinking about this further - we can fix this in the lifecycle, but I wonder if it makes sense to error (or at least warn) when the run image contains a duplicate top layer, because the resulting app image will not be rebased correctly. Looking here, we consider all layers underneath (and including) the first instance of the top layer diff ID to be the base image. The alternative would be to change how we do this in imgutil.
@jabrown85 do you have any thoughts here?
natalieparellano commented
@jericop thanks for this - I'm not able to reproduce 😕 - it does look like a different issue as I believe the error message is being emitted by pack. Is it possible you could open an issue on the pack repo?