denoland/deno_docker

Cache not caching all deps?

Maxim-Filimonov opened this issue · 10 comments

Maxim-Filimonov commented

Hi,
I noticed with following docker file:

ARG DENO_VERSION=1.30.0

FROM denoland/deno:bin-$DENO_VERSION AS deno
# debian based
FROM gcr.io/google.com/cloudsdktool/cloud-sdk:slim

COPY --from=deno /deno /usr/local/bin/deno

# # These steps will be re-run upon each file change in your working directory:
COPY deps.ts .
RUN deno cache deps.ts

ADD . .

RUN deno cache **/*.ts

ENTRYPOINT ["/usr/local/bin/deno"]
CMD ["run", "--cached-only", "--allow-env", "--allow-read", "--allow-write", "--allow-run", "--allow-net", "./src/http_server.ts"]

I get an error due to module missing in cache.

error: Uncaught TypeError: Specifier not found in cache: "https://deno.land/std@0.174.0/node/module_all.ts", --cached-only is specified.
    at async loadBuiltinNodeModules ([deno:cli/node/mod.rs:222:32]:2:25)

Not sure what's wrong quite new to deno. I do notice that if I remove --cached-only our image on cloud-run download deps on every request. Which is not something that we want.

kt3k commented

Sounds like a bug of Deno. https://deno.land/std@0.174.0/node/module_all.ts is the module used for enabling npm/node.js compatiblity.

The workaround would be adding the line RUN deno cache https://deno.land/std@0.174.0/node/module_all.ts after RUN deno cache **/*.ts

cc @bartlomieju

bartlomieju commented

This problem would be transparently solved by denoland/deno#16748, otherwise, we'd need to do some hacks to solve it. @dsherret any thoughts?

Maxim-Filimonov commented

Sounds like a bug of Deno. https://deno.land/std@0.174.0/node/module_all.ts is the module used for enabling npm/node.js compatiblity.

The workaround would be adding the line RUN deno cache https://deno.land/std@0.174.0/node/module_all.ts after RUN deno cache **/*.ts

cc @bartlomieju

That did help to certain extend. Got a new error now:

An npm specifier not found in cache: "ent"

What is strange I only get this error when building on Gcloud not when running a local docker build 🤔

kt3k commented

An npm specifier not found in cache: "ent"

What is strange I only get this error when building on Gcloud not when running a local docker build 🤔

Sounds like npm:ent is not cached for some reason only on Gcloud. What happens if you add RUN deno cache npm:ent command in your dockerfile?

Maxim-Filimonov commented

Apparently, there is some weird issue with Gcloud run. I don't know what does it do to docker image exactly but it completely ignores all the cache. The only thing I can think of is that .cache directory is used by pip too which in case of gcloud-sdk image might be something that google fiddles with at runtime.

I tried to download the same docker image that gcloud tries to run and it works perfectly without any cache issues on two different machines.
When tried to deploy that same instance to gcloud it fails miserably and cannot find any cache.

The workaround I have found is to bundle all dependencies using deno bundle inside docker image and run from .js file instead.
Probably, get the same result with less moving parts.

kt3k commented

I don't know what does it do to docker image exactly but it completely ignores all the cache. The only thing I can think of is that .cache directory is used by pip too which in case of gcloud-sdk image might be something that google fiddles with at runtime.

FYI you can control the path of cache directory by DENO_DIR env var. Setting it to some different path might also work around the issue

Maxim-Filimonov commented

I don't know what does it do to docker image exactly but it completely ignores all the cache. The only thing I can think of is that .cache directory is used by pip too which in case of gcloud-sdk image might be something that google fiddles with at runtime.

FYI you can control the path of cache directory by DENO_DIR env var. Setting it to some different path might also work around the issue

That worked!
I'm not sure why google cloud run doesn't like default caching directory but it seems to be the case. As soon as I switched to different directory using:

# set DENO_DIR to avoid conflicts with google cloud
ENV DENO_DIR=./.deno_cache

It works like it suppose to.

  • 👍1
kt3k commented

I just realized that Dockerfiles other than bin- version already set ENV and probably don't have this problem. Do you have any specific reason to use bin instead of ubuntu, alpine, etc?

inductor commented

Sorry to jump in! I suspected it might be caused that you're on root user, so I tried to change the exec user as nonroot.

It worked!

I confirmed this is working on Cloud Run. I do not have a fully explanation here because I couldn't find a specific doc about user limitation on Cloud Run (I'm a Google Developer Expert but I'm mainly active in GKE and Kubernetes community.) But can you take a look if this approach works for you? @Maxim-Filimonov

Btw ADD is not Dockerfile bast practice so I changed it to COPY :)

ARG DENO_VERSION=1.30.0

FROM denoland/deno:bin-$DENO_VERSION AS deno
# debian based
FROM gcr.io/google.com/cloudsdktool/cloud-sdk:slim

ARG USERNAME=deno
ARG GROUPNAME=deno
ARG UID=1001
ARG GID=1001
RUN groupadd -g $GID $GROUPNAME && \
    useradd -m -u $UID -g $GID $USERNAME
USER $USERNAME
WORKDIR /home/$USERNAME/app

COPY --from=deno /deno /usr/local/bin/deno

# # These steps will be re-run upon each file change in your working directory:
COPY deps.ts .
RUN deno cache deps.ts

COPY . .

RUN deno cache **/*.ts

EXPOSE 8000

ENTRYPOINT ["/usr/local/bin/deno"]
CMD ["run", "--cached-only", "--allow-env", "--allow-read", "--allow-write", "--allow-run", "--allow-net", "./src/http_server.ts"]
Maxim-Filimonov commented

I just realized that Dockerfiles other than bin- version already set ENV and probably don't have this problem. Do you have any specific reason to use bin instead of ubuntu, alpine, etc?

I used bin based on this example -> https://github.com/denoland/deno_docker#using-your-own-base-image