[Question] Why do you use an Operation System for build and another for production?
felinto-dev opened this issue · 15 comments
Hey there!
Using an Operation system for build and another for production, not can lead to problems with some specific binaries that were compiled for a non-alpine image?
That's why the base node_modules, which have been built up with the full OS (node:16) and have access to things like node-gyp and c++ bindings get passed over into the alpine build. Once it's built, they work fine, but building them in an alpine environment isn't possible, so to fix that you pre-build them, then copy them over.
Does that make sense? I haven't ran into an issue with bcrypt or pg with this setup, which is what I've built the template around (or rather, what I've used the template to eventually build)
I know this can cause issues with some NPM packages like ORM Prisma that need to install a specific binary for the OS.
I'll run some tests to see if it can actually cause a problem and let you know =)
Hey there!
Unfortunately I long I see this Dockerfile is incompatible with some libraries like Prisma. (I'm not aware of other libraries either)
Problem 1:
nest-docker-template/Dockerfile
Lines 15 to 21 in 1725ab3
Instead, to specify the exact directories/files you would copy, why don't do "COPY . ." + use .dockerignore?
In the case of Prisma, is necessary to copy the "prisma" directory. Because this Dockerfile means to be for "general use case", I can't suggest you do "COPY prisma...", instead, I suggest you use ".dockerignore".
PS: I am not sure about it but maybe you would run "yarn lint" before/after "yarn build". Don't copy eslint and prettier related files can lead to the impossibility for to do it.
Problem 2: Specific OS-related binaries
Still talking about Prisma, they install specific binaries related to the OS you're using.
Prisma Client depends on the query engine that is running as a binary on the same host as your application.
The query engine is implemented in Rust and is used by Prisma in the form of executable binary files. The binary is downloaded when prisma generate is called.
[...]
To solve this, if you know ahead of time that you will be deploying to a different environment, you can use the binary targets and specify which of the supported operating systems binaries should be included.
https://www.prisma.io/docs/guides/deployment/deployment-guides/deploying-to-a-different-os
https://www.prisma.io/docs/reference/api-reference/prisma-schema-reference#binarytargets-options
PS: I didn't ACTUALLY reproduce this issue because I had a lot of headaches with this third problem below 👇
Problem 3:
Prisma generate Typescript types and saves at "node_modules/.prisma/client" folder [link]
nest-docker-template/Dockerfile
Line 33 in 1725ab3
When you copy the node_modules folder from the "base" stage at Dockerfile you don't copy prisma generated files.
"prebuild": "rimraf dist && prisma generate",
I think this is the right place to put it because is not possible to compile TypeScript files with you make reference to Prisma generated types.
I don't see an easy way to solve these issues without changing the current Dockerfile "strategy". Currently, I am using this approach:
https://github.com/felinto-dev/bulk-downloader/blob/e87b0935875e70d8c80e72e32890efc904236243/Dockerfile
I can do a PR and basically try to "merge" the two strategies but I guess I will don't use a stage for "PROD, DEV, RELEASE (I guess)", as you do.
What do you think?
Instead, to specify the exact directories/files you would copy, why don't do "COPY . ." + use .dockerignore?
In the case of Prisma, is necessary to copy the "prisma" directory. Because this Dockerfile means to be for "general use case", I can't suggest you do "COPY prisma...", instead, I suggest you use ".dockerignore".
My general preference of being explicit over implicit. Using a .dockerignore would work just fine too.
PS: I am not sure about it but maybe you would run "yarn lint" before/after "yarn build". Don't copy eslint and prettier related files can lead to the impossibility for to do it.
Yeah, the yarn lint command would be ran before the build, but it's also commented out, so are these copy lines, so it's really not a problem. Just kind of showing it's possible. I've personally moved to doing linting and format checking in CI and outside of docker
I don't really have any experience with prisma, so I'm not sure what the docker workflow with it would and should look like. I created this template to be a good starter for building up docker images with Nest, hence the minimal set up and the different OSs for the build and prod stages. This dockerfile shows it's possible to end up with an image around 100MB as a final build size, instead of the 300MB+ that people were getting, but of course when adding in new libraries (like prisma) things can (and probably will) go wrong and tweaking will be necessary. If you'd like to add a second dockerfile like prisma.dockerfile or similar to show how prisma can be worked with, I'd be happy to add that, but I don't think the main dockerfile of this repo should be modified too much besides maybe removing the prettier and eslint configs and commands
My general preference of being explicit over implicit. Using a .dockerignore would work just fine too.
That's a good point, but like I say this Dockerfile means to be for "general use case", so being 100% explicit is kind of impossible, well... you understand my point neither.
This dockerfile shows it's possible to end up with an image around 100MB as a final build size, instead of the 300MB+ that people were getting, but of course when adding in new libraries (like prisma) things can (and probably will) go wrong and tweaking will be necessary.
I gotcha your point. I don't think we need to make a choice between size and flexibility. We can have both.
If you'd like to add a second dockerfile like prisma.dockerfile or similar to show how prisma can be worked with, I'd be happy to add that, but I don't think the main
I don't think is a good approach. The problem is not generated by special requirements by Prisma, but because use two different OS for building and another for production. This is not a Prisma related issue.
Does that make sense? I haven't ran into an issue with bcrypt or pg with this setup, which is what I've built the template around (or rather, what I've used the template to eventually build)
Could you give me instructions on how could I reproduce that? Maybe tweaking the image to support both use cases is a better approach than creating N-Dockerfiles for specific libraries in NPM but I would give it a try and check if is possible first.!
you understand my point neither.
No, I understood your point, and mentioned that I rather be explicit. This is a template that is to be used, modified, and customized to each user's needs. It generally works out of the box. Perhaps we could add more to the README about customization of it, but I'd personally rather not set up the .dockerignore and then get people come screaming "Why doesn't it work?" because something was ignored that they needed. If it's explicit, and people see what is being pulled in, I believe they can better understand how to customize it.
I don't think is a good approach. The problem is not generated by special requirements by Prisma, but because use two different OS for building and another for production. This is not a Prisma related issue.
To my knowledge, prisma is the first image I've heard of that this approach might not work on. Using mutli-stage builds with alpine as the final image base is rather common honestly. Even Docker's docs website shows doing such a thing.
Could you give me instructions on how could I reproduce that?
I've made a sample here using bcrypt to show how the bcrypt package getting built on the regular node:16 image and being used on node:16-alpine works.
Using mutli-stage builds with alpine as the final image base is rather common honestly. Even Docker's docs website shows doing such a thing.
Both links are related to Go. I guess the Go language does not care much about what OS you use rather than some libraries like Prisma and bcrypt care.
I've made a sample here using bcrypt to show how the bcrypt package getting built on the regular node:16 image and being used on node:16-alpine works.
Thanks for your time! I will do some tests and let you know If I get some insights about how we could address this issue.
It generally works out of the box. Perhaps we could add more to the README about customization of it, but I'd personally rather not set up the .dockerignore and then get people come screaming "Why doesn't it work?" because something was ignored that they needed.
I could argue the same thing about the current approach. Basically, the goal should be to choose the approach that is the best user-friendly, simple, and flexible. I will investigate it further and let you know If there is something we can do to improve it.
Both links are related to Go. I guess the Go language does not care much about what OS you use rather than some libraries like Prisma and bcrypt care.
I suppose so. I've not run into an issue withg this approach with pg or with bcrypt like I've mentioned before, but now that I'm trying to find other examples using node and node-alpine I'm not able to. Perhaps the base should be changed to node-alpine and have the necessary additions brought in for these built libraries (node-gyp, etc).
Basically, the goal should be to choose the approach that is the best user-friendly, simple, and flexible
I believe we have the same goal here, just different views on how it should be approached. I will say the other reason I like using explicit COPY commands vs a general dockerignore is for the ease of cache-busting. If we only have COPY . . and we need to ensure we've updated the .dockerignore, does the docker engine know about the now non-ignore file, or does it read the cache and say "Well, done this before"? With the explicit COPY file ./location and addition or removal of files the COPY command's hash is now different and the cache is definitely invalidated, so the docker engine for sure runs the step
I've made a sample here using bcrypt to show how the bcrypt package getting built on the regular node:16 image and being used on node:16-alpine works.
I can't reproduce the issue.
EDITED Dockerfile:
FROM node:16-alpine as base
WORKDIR /app
COPY package.json \
yarn.lock \
./
RUN yarn --production
# lint and formatting configs are commented out
# uncomment if you want to add them into the build process
FROM base AS dev
COPY nest-cli.json \
tsconfig.* \
# .eslintrc.js \
# .prettierrc \
./
# bring in src from context
COPY ./src/ ./src/
RUN yarn
# RUN yarn lint
RUN yarn build
# use one of the smallest images possible
FROM node:16-alpine
# get package.json from base
COPY --from=base /app/package.json ./
# get the dist back
COPY --from=dev /app/dist/ ./dist/
# get the node_modules from the intial cache
COPY --from=base /app/node_modules/ ./node_modules/
# expose application port
EXPOSE 3000
# start
CMD ["node", "dist/main.js"]
I only change the base image to "node:16-alpine" and remove "node-prune" for testing purposes.
curl http://localhost:3000
$2b$12$OYCsx.s12e0tUq05lULn6OkFBDqRCe3WKiEusc.AG0pDsi6ojDNmm%
curl... get the same result in both images.
I can't run the command:
curl -X http://localhost:3000 -d 'originalValue=hello world!&hash=<hash from first request>'
curl: no URL specified!
curl: try 'curl --help' or 'curl --manual' for more information
I don't have much experience using the curl command. Could you check?
Ah, my apologies on that. I use xh personally and thought I remembered the full curl command. It's actually curl -X POST http://...
If we only have COPY . . and we need to ensure we've updated the .dockerignore, does the docker engine know about the now non-ignore file, or does it read the cache and say "Well, done this before"? With the explicit COPY file ./location and addition or removal of files the COPY command's hash is now different and the cache is definitely invalidated, so the docker engine for sure runs the step
Did you actually already fancy this issue? Is the first time I heard about it.
I can do some tests to check if is still a problem. Sounds like a bug in an earlier version of Docker or someone who did something crazy like to put .dockerignore inside .dockerignore or so.
Well, we can check this another issue first 👆
Okay, so it looks like COPY . . does still get cache busted if the .dockerignore is changed, so that alleviates that concern of mine. Probably just a docker noob concern of mine that I never fully tested. I guess then it's just a question of what is more user friendly, a COPY . . or a COPY ...files ./?
No matter the approach you use, you always need to double-check .dockerignore because, for performance and best practices, you should ignore the node_modules and dist folder at least.
As much I understand your concern for copying only what is necessary and being explicit about what you would have at the final Docker image, no matter who is the user is quite important to check the .dockerignore anyway.
For most users, only replicating the .gitignore file is enough.
There are other situations where this current approach does not will works, as monorepo too.
I suggest doing something like it:
# You MUST specify files/directories you don't want on your final image like .env file, build, etc. The file ../.dockerignore is a good starting point.
COPY . .
RUN ls -l
I will let some friends know about this issue, maybe can be interesting to get another point of view, either :-)
Did you were able to replicate the issue about not being able to install "bcrypts" on the alpine-based image? I can't.
Did you were able to replicate the issue about not being able to install "bcrypts" on the alpine-based image? I can't.
No, and upon further investigation, it seems this was something that happened back in Node 10, which is probably about when I wrote this (and even then I'm having a problem replicating this, so I wonder if there was a specific bug that eventually got patched). I'm happy to have the base image updated to alpine as well so there's not two OSs being used.
no matter who is the user is quite important to check the
.dockerignoreanyway.
Completely agree here, though we both know that there are those who won't know to do it, or won't care to do it.
For most users, only replicating the
.gitignorefile is enough.
True enough.
There are other situations where this current approach does not will works, as monorepo too.
Yeah, monorepos are another beast all together, and this template will definitely not be enough for them.
I suggest doing something like it:
# You MUST specify files/directories you don't want on your final image like .env file, build, etc. The file ../.dockerignore is a good starting point.
COPY . .
RUN ls -l
I'd be happy to see this update in a PR after our conversation here
I'd be happy to see this update in a PR after our conversation here
I will do :-)