Run image extension does not install RPMs that are already installed in the builder
Opened this issue · 5 comments
yters commented
Summary
When the run image run.Dockerfile is extended to RUN dnf install RPMs, if the RPM is already in the builder image then it is not installed in the run image. This is a problem since the run image does not have access to the RPMs in the builder image.
Reproduction
Steps
- Create a build image with the following Dockerfile named
<path>/build-base-python3.11:debug.
FROM registry.access.redhat.com/ubi8-minimal:8.8
ARG STACK_ID="debug"
ENV CNB_USER_ID=${CNB_UID:-1000}
ENV CNB_GROUP_ID=${CNB_GID:-1000}
RUN microdnf update
RUN microdnf install python3.11
RUN groupadd cnb --gid ${CNB_GROUP_ID}
RUN useradd --uid ${CNB_USER_ID} --gid ${CNB_GROUP_ID} -m -s /bin/bash cnb
USER cnb
LABEL io.buildpacks.stack.id=${STACK_ID}
ENV CNB_STACK_ID=${STACK_ID}
ENV CNB_USER_ID=${CNB_UID:-1000}
ENV CNB_GROUP_ID=${CNB_GID:-1000}
- Create a run image with the following Dockerfile named
<path>/run-base:debug
FROM registry.access.redhat.com/ubi8-minimal:8.8
ARG STACK_ID="debug"
ENV APP_USER_ID=${APP_UID:-31460}
ENV APP_GROUP_ID=${APP_GID:-31460}
LABEL io.buildpacks.stack.id=${STACK_ID}
ENV CNB_STACK_ID=${STACK_ID}
- Create an extension with the following files, named
<path>/extension-dnf-install:debug.
extension.toml
api = "0.10"
[extension]
id = "dnf-install"
name = "dnf python3.11 RPM installer"
description = "Extension that installs python3.11 RPM in the run image."
version = "debug"
[[targets]]
os = "linux/amd64"
arch = "amd64"
generate/run.Dockerfile
ARG base_image
FROM ${base_image}
USER root
RUN microdnf install python3.11
ARG user_id
USER \${user_id}
- Create a buildpack from the
hello-wordexample buildpack, named<path>/buildpack-hello-world:debug. - Create a builder with the following
builder.toml, named<path>/builder-python3.11:debug.
[[extensions]]
uri = "<path>/extension-dnf-install:debug"
id = "dnf-install"
[[order-extensions]]
[[order-extensions.group]]
id = "dnf-install"
version = "debug"
[[buildpacks]]
uri = "<path>/buildpack-hello-world:debug"
id = "hello-world"
[[order]]
[[order.group]]
id = "hello-world"
version = "debug"
[stack]
id = "debug"
run-image = "<path>/run-base:debug"
build-image = "<path>/build-base-python3.11:debug"
- Use the lifecycle phases in the
builder-python3.11:debugimage to create an image. Create the following files in the current directory.
config/run.toml
[[images]]
image = "<path>/run-base:debug"
source/hello.sh
#!/bin/sh
echo hello
Then, run the following commands to build a container.
export image=<path>/builder-python3.11:debug
docker run --user root -v ~/.docker:/workspace/dockerconfig -v $(pwd)/source:/workspace/source -v $(pwd)/config:/workspace/config -it --rm $image /bin/bash
# These following commands are run within the container started in the immediately previous `docker run` command.
mkdir /kaniko
export run_image=<path>/stack-rhel8-run/amd64:debug
export image=<path>/test:debug
DOCKER_CONFIG=/workspace/dockerconfig CNB_PLATFORM_API=0.12 /cnb/lifecycle/analyzer -cache-image=$image -uid=1000 -gid=1000 -run-image $run_image $image
su cnb
CNB_PLATFORM_API=0.12 CNB_EXPERIMENTAL_MODE=silent /cnb/lifecycle/detector -app=/workspace/source/ -run=/workspace/config/run.toml
exit
DOCKER_CONFIG=/workspace/dockerconfig CNB_PLATFORM_API=0.12 /cnb/lifecycle/restorer -cache-image=$image -uid=1000 -gid=1000
mkdir /layers/extended
DOCKER_CONFIG=/workspace/dockerconfig CNB_PLATFORM_API=0.12 CNB_EXPERIMENTAL_MODE=silent /cnb/lifecycle/extender -app=/workspace/source -kind=run -extended=/layers/extended
You'll see a result like this.
$ export image=<path>/builder-python3.11:debug
$ docker run --user root -v ~/.docker:/workspace/dockerconfig -v $(pwd)/source:/workspace/source -v $
(pwd)/config:/workspace/config -it --rm $image /bin/bash
[root@102b97e9b6a4 layers]# mkdir kaniko
[root@102b97e9b6a4 layers]# rm -rf kaniko/
[root@102b97e9b6a4 layers]# mkdir /kaniko
[root@102b97e9b6a4 layers]# export run_image=<path>/stack-rhel8-run/amd64:debug
[root@102b97e9b6a4 layers]# export image=<path>/test:debug
[root@102b97e9b6a4 layers]# DOCKER_CONFIG=/workspace/dockerconfig CNB_PLATFORM_API=0.12 /cnb/lifecycle/analyzer -cache-image=
$image -uid=1000 -gid=1000 -run-image $run_image $image
Timer: Analyzer started at 2024-02-06T17:49:04Z
Image with name "<path>/test:debug" not found
Timer: Analyzer ran for 1.438062316s and ended at 2024-02-06T17:49:05Z
[root@102b97e9b6a4 layers]# su cnb
[cnb@102b97e9b6a4 layers]$ CNB_PLATFORM_API=0.12 CNB_EXPERIMENTAL_MODE=silent /cnb/lifecycle/detector -app=/workspace/source/
-run=/workspace/config/run.toml
Warning: Platform requested experimental feature 'Dockerfiles'
Timer: Detector started at 2024-02-06T17:49:11Z
dnf-install debug
hello-world debug
Timer: Detector ran for 4.60913ms and ended at 2024-02-06T17:49:11Z
Timer: Generator started at 2024-02-06T17:49:11Z
Timer: Generator ran for 841.845µs and ended at 2024-02-06T17:49:11Z
[cnb@102b97e9b6a4 layers]$ exit exit
[root@102b97e9b6a4 layers]# DOCKER_CONFIG=/workspace/dockerconfig CNB_PLATFORM_API=0.12 /cnb/lifecycle/restorer -cache-image=
$image -uid=1000 -gid=1000
Timer: Restorer started at 2024-02-06T17:49:21Z
Layer cache not found
Timer: Restorer ran for 1.114690805s and ended at 2024-02-06T17:49:22Z
[root@102b97e9b6a4 layers]# mkdir /layers/extended
[root@102b97e9b6a4 layers]# DOCKER_CONFIG=/workspace/dockerconfig CNB_PLATFORM_API=0.12 CNB_EXPERIMENTAL_MODE=silent /cnb/lif
ecycle/extender -app=/workspace/source -kind=run -extended=/layers/extended
INFO[0000] Built cross stage deps: map[]
INFO[0000] Executing 0 build triggers
INFO[0000] Building stage 'base@sha256:79db881f1fa031623c738a9f0cf7a3dce4ceb2e0183c17d6364bd5b3cf0b3f02' [idx: '0', base-idx:
'-1']
INFO[0000] Cmd: USER
INFO[0000] Checking for cached layer oci:/kaniko/cache/layers/cached:eba5636260b56b6f0c94d54b9a764036297f933fa03d198068c41c6e
723c055a...
INFO[0000] No cached layer found for cmd RUN microdnf install python3.11
INFO[0000] Cmd: USER
INFO[0000] Unpacking rootfs as cmd RUN microdnf install python3.11 requires it.
INFO[0000] Skipping unpacking as no commands require it.
INFO[0000] USER root
INFO[0000] Cmd: USER
INFO[0000] No files changed in this command, skipping snapshotting.
INFO[0000] RUN microdnf install python3.11
INFO[0000] Initializing snapshotter ...
INFO[0000] Taking snapshot of full filesystem...
INFO[0000] Cmd: /bin/sh
INFO[0000] Args: [-c microdnf install python3.11]
INFO[0000] Util.Lookup returned: &{Uid:0 Gid:0 Username:root Name:root HomeDir:/root}
INFO[0000] Performing slow lookup of group ids for root
INFO[0000] Running: [/bin/sh -c microdnf install python3.11]
(microdnf:72): librhsm-WARNING **: 17:49:30.110: Found 0 entitlement certificates
(microdnf:72): librhsm-WARNING **: 17:49:30.111: Found 0 entitlement certificates
Nothing to do.
INFO[0000] Taking snapshot of full filesystem...
INFO[0000] ARG user_id
INFO[0000] Pushing layer oci:/kaniko/cache/layers/cached:eba5636260b56b6f0c94d54b9a764036297f933fa03d198068c41c6e723c055a to
cache now
INFO[0000] No files changed in this command, skipping snapshotting.
INFO[0000] USER \${user_id}
INFO[0000] Cmd: USER
INFO[0000] No files changed in this command, skipping snapshotting.
INFO[0000] Skipping push to container registry due to --no-push flag
Warning: The original user ID was app but the final extension left the user ID set to ${user_id}.
Timer: Extender ran for 965.770627ms and ended at 2024-02-06T17:49:30Z
Current behavior
In the extend phase when it comes time to install python3.11 it will say:
INFO[0000] Running: [/bin/sh -c microdnf install python3.11]
Nothing to do.
despite python3.11 not being installed in the run image.
Expected behavior
I expect python3.11 to be installed in the run image since it is not installed in the base run image.
Context
lifecycle version
Lifecycle version is 0.17.2.
platform version(s)
$ pack report
Pack:
Version: 0.32.1+git-b14250b.build-5241
OS/Arch: linux/amd64
Default Lifecycle Version: 0.17.2
Supported Platform APIs: 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12
Config:
experimental = true
lifecycle-image = "buildpacksio/lifecycle:0.17.1"
layout-repo-dir = "/home/<redacted>/.pack/layout-repo"
$ docker info
Client: Docker Engine - Community
Version: 25.0.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.12.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.24.2
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 35
Running: 1
Paused: 0
Stopped: 34
Images: 51
Server Version: 25.0.1
Storage Driver: overlay2
Backing Filesystem: btrfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
runc version: v1.1.11-0-g4bccb38
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.6.13-100.fc38.x86_64
Operating System: Fedora Linux 38 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 31.03GiB
Name: fedora
ID: 6a61b820-9dc5-4e48-9607-b2df128276e9
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
natalieparellano commented
@yters looking at your code above, I think the issue is that you are trying to extend the run image in the context of the build image. So in this snippet:
export image=<path>/builder-python3.11:debug
docker run --user root -v ~/.docker:/workspace/dockerconfig -v $(pwd)/source:/workspace/source -v $(pwd)/config:/workspace/config -it --rm $image /bin/bash
...
Where $image is the builder image - there is only one docker run instruction. There should be at least three (one for analyze/detect/restore/extend-build that uses $image, one for extend-run that uses $run_image, and one for export AFTER both extend phases have completed that uses $image). You would mount the same directories across all three invocations. If it's helpful here is how we do it in pack.
You may be interested in this (soon to be implemented) RFC which would make it easier to share things across the builder image and the run image: buildpacks/rfcs#301
yters commented
Thanks for the response.
The docker run starts a terminal inside the builder docker container, and the lifecycle phase commands are run inside the container. I don't see how using the run image instead would work, since the run image doesn't have the lifecycle commands:
$ docker run --user root -it --rm <path>/run-base:debug /bin/sh
sh-4.4# ls /cnb/lifecycle
ls: cannot access '/cnb/lifecycle': No such file or directory
If I were to use the pack command to extend a run image instead of the lifecycle phase commands, should this work? I.e. the RPMs in the builder image will not conflict with the run image?
natalieparellano commented
the run image doesn't have the lifecycle commands
Ah yes, you are right. In the case of pack we download the lifecycle layer from the "lifecycle image" and append that to the run image before performing run image extension. You could accomplish something similar with volume mounts for local testing.
yters commented
Thanks, that's a good idea. I'll test it out, and close this ticket if successful.
natalieparellano commented
Any further update here? Can we close this issue?