S3 Storage Fails with storage: read exceeds limit
mhkarimi1383 opened this issue · 8 comments
Description
We are building an image based on php:8.2-cli-bookworm (using earthly)
and our registry storage is S3, and we are able to push image without problem, but when we want to pull image we are getting
500: Internal Server error, with unknown error in log and in details we are getting storage: read exceeds limit
Reproduce
Here is my Earthfile
VERSION 0.8
ARG --required PHP_VERSION
ARG --required DEBIAN_VERSION
FROM docker.io/library/php:${PHP_VERSION}-cli-${DEBIAN_VERSION}
WORKDIR /var/www/html/
ENV DEBIAN_FRONTEND=noninteractive
ENV LC_ALL=C.UTF-8
ENV TERM=xterm-color
ENV COMPOSER_ALLOW_SUPERUSER 1
ENV PATH /var/www/html/vendor/bin:/composer/vendor/bin:$PATH
RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
reqs:
RUN set -eux; \
apt-get update; \
apt-get install -y apt-transport-https; \
pecl channel-update pecl.php.net \
&& apt-get install -yqq --no-install-recommends \
apt-utils \
gnupg \
gosu \
git \
curl \
ca-certificates \
supervisor \
libmemcached-dev \
libz-dev \
libbrotli-dev \
libc-ares-dev \
libpq-dev \
libjpeg-dev \
libpng-dev \
libfreetype6-dev \
libssl-dev \
libwebp-dev \
libmcrypt-dev \
libonig-dev \
libzip-dev zip unzip \
libargon2-1 \
libidn2-0 \
libpcre2-8-0 \
libpcre3 \
libxml2 \
libzstd1 \
procps \
libcurl4-openssl-dev
RUN apt-get install -y --no-install-recommends vim nano unzip net-tools \
libmagickwand-dev libcurl4-gnutls-dev libgmp-dev libxslt1-dev zlib1g-dev libicu-dev g++
RUN pecl install -f \
igbinary
RUN docker-php-ext-enable \
igbinary
RUN docker-php-ext-install \
bz2 \
calendar \
curl \
exif \
gettext \
gmp \
iconv \
pdo \
pdo_pgsql \
shmop \
soap \
sysvmsg \
sysvsem \
sysvshm \
xml \
xsl
RUN php -m
RUN set -xe; \
docker-php-ext-configure zip \
&& docker-php-ext-install zip \
&& php -m | grep -q 'zip'; \
docker-php-ext-install \
mbstring \
pdo_mysql; \
docker-php-ext-configure gd \
--prefix=/usr \
--with-jpeg \
--with-webp \
--with-freetype \
&& docker-php-ext-install gd; \
php -r 'var_dump(gd_info());'
RUN docker-php-ext-install opcache
RUN pecl install -o -f redis && \
rm -rf /tmp/pear && \
docker-php-ext-enable redis
RUN docker-php-ext-install pcntl
RUN docker-php-ext-install bcmath
RUN pecl install -D 'enable-openssl="yes" enable-http2="yes" enable-swoole-curl="yes" enable-cares="yes"' swoole
RUN docker-php-ext-enable swoole && \
php -m | grep -q "swoole"
RUN docker-php-ext-configure intl && \
docker-php-ext-install intl
ARG INSTALL_PG_CLIENT=true
ARG INSTALL_POSTGIS=true
IF [ ${INSTALL_PG_CLIENT} = true ]
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
apt-get install -yqq gnupg \
&& . /etc/os-release \
&& echo "deb http://apt.postgresql.org/pub/repos/apt $VERSION_CODENAME-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
&& curl -sL https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - \
&& apt-get update -yqq \
&& apt-get install -yqq --no-install-recommends --show-progress postgresql-client-16 postgis;
IF [ ${INSTALL_POSTGIS} = true ]
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
--mount=type=cache,target=/var/lib/apt,sharing=locked \
apt-get install -yqq --no-install-recommends --show-progress postgis;
END
END
RUN curl -s https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin/ --filename=composer
RUN composer global require bamarni/symfony-console-autocomplete
RUN curl -sL https://deb.nodesource.com/setup_20.x | bash - \
&& apt-get install -y nodejs \
&& npm install -g npm \
&& curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
&& echo "deb https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list \
&& apt-get update \
&& apt-get install -y yarn
docker:
FROM +reqs
ARG --required CONTAINER_IMAGE_PREFIX
ARG --required PHP_VERSION
ARG --required DEBIAN_VERSION
SAVE IMAGE --cache-hint --push ${CONTAINER_IMAGE_PREFIX}/php:${PHP_VERSION}-cli-${DEBIAN_VERSION}
Expected behavior
Pull image without problem
registry version
image tag: registry:2
version: registry github.com/docker/distribution 2.8.3
Additional Info
Image size is about 600 MBs
But there was nothing in directories related to this image/manifest in my S3 bucket
and
There is no problem while using Dockerfile (I think there are some problems with buildkit/earthly)
There is a read limit on pulls to prevent memory allocations (
distribution/registry/storage/io.go
Line 13 in 5d5c60f
docker pull?Both using docker pull and FROM keyword, and also K8s pull using ContainerD
Also I have a worker that keeps my images up to date based on policies it will check for updates every one minute per image, but why switching to Dockerfile fixed problem?
I have searched for the error message (after storage: prefix but found nothing)
Thank you for sharing the code
As I see on code I think I was trying to make the image to be a single layer So it was exceeding that limit
I would also recommend you to switch to v3 release -- it has a lot of patches applied and a bunch of performance improvements. Consider v2.x release to be in maintenance mode at this point
Thanks
Is that stable?
Also what are changes?
I saw nothing in docs about that
It's not been marked as stable yet, but the changes have been accumulated over the past couple of years -- before we released v3 there had been no release for over 5 years. See the not-fully complete changelog here: https://github.com/distribution/distribution/releases/tag/v3.0.0-alpha.1
Thank you again
I have some problems with garbage collection
That when I run that it will not remove things completely and the removed tag is listed in the tag list api
Also I have some integrations with HTTP V2 API and Debug API
And I need a lot of tests and more list of deprications on API, etc.