Multiple levels of wildcards not supported or not working properly
csMACnz opened this issue · 15 comments
I used to be able to copy all files of a specific type from two-levels nested. This no longer works in BuildKit.
Why? To avoid cache busting. We want to restore our dependencies from all *.csproj files, and not invalidate that cache layer when a source file changes.
see https://stackoverflow.com/a/58697043 as the defacto approach most dotnet docker builds will be using.
If we just copy everything over first, that means any change triggers cache-busting and a full package restore is again required. Without wildcards, we have to manually maintain a long list of csproj file COPYs that is error-prone and unmaintainable across many solutions.
# These all worked without buildkit but now fail with buildkit
COPY src/*/*.csproj ./
COPY */*/*.csproj ./
This puts all found files into a single folder (which at least works for the goal we are trying to achieve)
I now get errors like:
error from sender: readdir: open *: The filename, directory name, or volume label syntax is incorrect.
or sometimes nothing is copied, so i get missing files errors later in the next RUN command.
As far as I an tell, COPY */*.csproj ./ still works as expected, when csproj files are only one-level nested, but doesn't work for the two-level nested case.
As a workaround, I might be able to move to a one-level nested structure, though ideally i wouldn't have to change my structure to fit docker for something that used to work?
Thanks!
I'm having the same problem while configuring buildkit for same caching purpose in .NET. Is there an alternative ?
COPY */*.csproj ./ (with one-level nested) not working either, is there a tricks ?
Thanks 😊
I don't think it is about multiple level of wildcards. When I run the below script, that has only one level of wildcard, it fails:
mkdir project-x-dependencies
echo "line-of-text" > project-x-dependencies/file1.txt
echo FROM busybox > Dockerfile
echo COPY project-*-dependencies/file1.txt /work/ >> Dockerfile
docker build .
It appears to me that it is about wildcards in directories.
"needs investigation"
Any update on this? after the last update, no one in our team using the windows version of docker can build their images anymore :/
EDIT:
Fellow googlers (temp) fix can be found here
moby/moby#42133 (comment)
Good morning,
in our case we have smth like:
COPY ./drivers/build/3rdparty-jdbc-drivers/**/* jars/3rdparty/
and with "buildkit": true it doesn't work
If this is gonna be fixed, it would be nice have a way to have these wildcards and preserve the folder structure as well.
Same issue happened while I was trying to build the - https://github.com/Squidex/squidex.
After setting "buildkit": false was able to build but resetting "buildkit": true its giving an error from sender: readdir: open backend\extensions\*: The filename, directory name, or volume label syntax is incorrect..
Same with
COPY src/dotnet/*/*.csproj ./
doesn't work with buildkit, works fine without buildkit
Please fix this - it's creating so much work and can't be that difficult to solve this - please. IMHO this ticket doesn't need any investigation, it needs solving :D (talking of labels)
Stumbled into a workaround thanks to a very unique combination of sheer laziness and force of stubborn will.
I just added a "staging" layer and slammed my entire context to copy from It appears to have the added benefit of standardizing to *nixy / paths from the get-go, and just copying in --from=ctx instead:
FROM scratch as ctx
WORKDIR /app
COPY . /app
FROM mcr.microsoft.com/dotnet/sdk:6.0 as build
COPY --from=ctx "app/src/*/*.csproj" "./"
I'm sure this is bloating my caches with unneeded builds though. I'm trying, but I can't figure out how to explicitly exclude the stage from caching (via Dockerfile at least).
Is there anything we can do to get this looked at?
Is there any hope if this issue being addressed?
Hi all
This seems like a very strange issue that globbing does not work?
Is this fo for buildkit on windows only (via WSL)
The proposed work arounds above defeat the purpose of layer caching. in docker build?
OK so its not a host issue.. its just globbing is not implemented in copy for buildkit
That is very strange to have feature parity missing like that.. especially when the legacy builder is about to be removed.
Who do we escalate this too because I have noticed buildkit issues have been open for years and nobody really doing anything here?
Any updates on this?