Should the containerd-config.toml file have discard_unpacked_layers set to true by default

[pptb-aws]

Environment:

• AWS Region: All
• Instance Type(s): All
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): Any version using containerd
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): Any version using containerd
• AMI Version: Any version using containerd
• Kernel (e.g. uname -a): NA
• Release information (run cat /etc/eks/release on a node):

BASE_AMI_ID="ami-044dcf122cb58da69"
BUILD_TIME="Mon Jul  3 03:32:16 UTC 2023"
BUILD_KERNEL="5.4.247-162.350.amzn2.x86_64"
ARCH="x86_64"

Issue:

The issue here is that the configuration used for containerd can result in significantly increased disk usage because containerd will store both the packed and unpacked layers which in the case where the image does not compress well as a tarball can result in increased utilization of the root volume. To see this in action you can used these commands [1] to make a dockerfile as the random data will not compress well.

The workaround is to create your own containerd config file then add discard_unpacked_layers = true to the plugins."io.containerd.grpc.v1.cri".containerd section. Then pass this to the bootstrap command using the --containerd-config-file after putting the file on the instance, usually by downloading from S3 although a custom AMI could also be used. However, this seems like a lot of work just to bring containerd in line with the space usage of docker. Which bring us to the question, should the containerd-config.toml file have discard_unpacked_layers set to true by default?

[1]

mkdir /var/lib/docker/cstest
cd /var/lib/docker/cstest
dd if=/dev/urandom of=testfile.img bs=$((1024*1024)) count=1007
dd if=/dev/urandom of=testfile2.img bs=$((1024*1024)) count=2015
dd if=/dev/urandom of=testfile3.img bs=$((1024*1024)) count=2015
cat > Dockerfile <<EOF
FROM ubuntu
COPY ./testfile.img /testfile.img
COPY ./testfile2.img /testfile2.img
COPY ./testfile3.img /testfile3.img
CMD ["/bin/bash","cat testfile.img"]
EOF

[cartermckinnon]

I think it makes sense to enable this by default. My understanding is you'd only need the unpacked layers if you plan to push the image later on; which shouldn't be a common case on a Kubernetes node. I'll run this by the container runtime team for a sanity check 👍