As a Developer I want to be able to build Windows Containers with BuildKit

Question

As a Developer I want to be able to build Windows Containers with BuildKit

bplasmeijer opened this issue 4 years ago · 134 comments

bplasmeijer commented 4 years ago

The windows BuildKit is only supported on Linux

fady-azmy-msft commented 2 years ago

FYI @Muzz

Answer 1 · 2020-07-02T20:52:56.000Z

It is possible to use buildx (which uses buildkit) if there are no RUN commands in the docker file:

docker buildx create --name img-builder --use
docker buildx build --no-cache --pull --push \
  --platform "windows/amd64" -t $imagetag -f $dockerfile .
docker buildx rm img-builder

ARG nanoserverTag="mcr.microsoft.com/windows/nanoserver:1809-amd64"
# othertags list: curl -L https://mcr.microsoft.com/v2/windows/nanoserver/tags/list

FROM ${nanoserverTag}
COPY some.exe "/Windows/System32/some.exe"

It would be interesting to see if could use some windows emulator connected to Binfmt_misc as QEMU is used for other architectures

Answer 2 · 2020-07-07T15:30:13.000Z

There is an issue logged on BuildKit repo moby/buildkit#616

Answer 3 · 2020-07-08T05:17:35.000Z

@jsturtevant If you wanted to go down that path, I believe Wine can be used with binfmt_misc, although I'm not sure how well that would work with namespaces since there's a WineServer and configuration directory to create, and you probably don't want that in the final image...

It would be easier to prototype that on a raw Linux box than via the Linux support in Docker on Windows, I think.

Answer 4 · 2020-09-17T16:01:07.000Z

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

Answer 5 · 2020-09-17T16:43:46.000Z

Current status of my work on moby/buildkit#616 is that it's stalled waiting on a pile of 4 containerd PRs to be reviewed and advanced. The top of the stack is containerd/containerd#4419, but there's some outstanding design questions on the top two PRs regarding HCS behaviours, see microsoft/hcsshim#853, but the bottom two are just pending containerd maintainer review to merge or rework as appropriate.

And after that, will come fixing path-related issues in the BuildKit LLB layer, for which I'd only scratched the surface of in moby/buildkit#1621, exploring the solution space, and which needs to be reworked per maintainer feedback.

Answer 6 · 2020-10-19T16:00:30.000Z

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

Answer 7 · 2020-10-20T02:27:16.000Z

Status unchanged, still waiting on merging or feedback on containerd support for Windows container creation/export/mount support, and then continuing the work on BuildKit itself, starting from moby/buildkit#1621

Answer 8 · 2020-10-22T00:21:50.000Z

@kevpar Do you have any information on the containerd PRs mentioned above?

Answer 9 · 2020-11-21T16:00:18.000Z

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

Answer 10 · 2020-12-22T16:00:17.000Z

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

Answer 11 · 2020-12-22T18:02:11.000Z

Still working through containerd PRs per above. One of four has landed, and one needs to be rewritten or at least iterated upon further. I've also opened PR on hcsshim again to remove some duplication of code with the containerd PRs.

That said, Docker is looking at supporting containerd on Windows for 21.x (moby/moby#41455), which might mean we get BuildKit on Docker+containerd even without that containerd PR stack, as the parts I was trying to implement in containerd (WCOW layer management) are provided by Docker when Docker uses containerd, as far as I know.

I haven't looked closely at how BuildKit integrates with Docker though, it's possible it'll still need fixes in containerd, and either way, BuildKit itself has higher-level issues to be resolved, e.g., moby/buildkit#1621.

Answer 12 · 2021-01-22T16:00:18.000Z

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

Answer 13 · 2021-01-22T22:45:36.000Z

Not much has changed since December. The containerd PRs are now in a reasonable state, two are ready-to-go and one (containerd/containerd#4419) is functional but possibly needs some discussion about the test suite changes, and depends on a hcsshim feature PR (microsoft/hcsshim#901).

Some recent testing suggests that that hcsshim PR might have issues with Windows 10 20H2 that don't appear on Windows Server LTSC2019, but I haven't definitely proved that as I have only tested the latter via CI pipelines so far. Edit: Still using CI pipelines, I confirmed that the PR works on RS5 on AppVeyor (VS 2019 image), but fails on my personal Windows 10 20H2 box, suggesting that some part of the reverse engineering used to build that PR was incorrect or incomplete.

Answer 14 · 2021-02-03T22:47:38.000Z

@weijuans-msft can you provide clarity here as part of your effort to provide containerd parity with Docker developer experience.

Answer 15 · 2021-03-06T16:01:13.000Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

Answer 16 · 2021-03-07T01:12:10.000Z

Compared to January, I've identified and have a fix for the issue that microsoft/hcsshim#901 shows on Windows 10 20H2, but that PR is still pending merge, and then the fix will be up for PR.

The containerd PRs (containerd/containerd#4399, containerd/containerd#4415, and containerd/containerd#4419) should all be in a reasonable state to flow through, although the latter depends on microsoft/hcsshim#901, often triggers a now-known containerd issue (containerd/containerd#4924) in its test suite, and also frequently triggers an unknown (hcsshim?) issue that I've yet been unable to replicate outside containerd.

It's possible that the Docker/containerd integration for Windows (moby/moby#41455 is the best place to track that) will mean some or all of those PRs stop being critical-path for BuildKit on Windows, and then the BuildKit-local platform issues (i.e. filesystem handling and similar) can be addressed.

Answer 17 · 2021-03-31T18:03:43.000Z

@TBBle great work here! I changed this to "In Progress" as I am seeing you and lots of others in the moby and containerd community working on this. Thank you.

Answer 18 · 2021-05-01T16:01:08.000Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

Answer 19 · 2021-06-01T16:01:04.000Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

Answer 20 · 2021-07-02T16:01:00.000Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

Answer 21 · 2021-07-22T15:57:27.000Z

Without this, is there any good way to handle build-time secrets for Windows containers?

Answer 22 · 2021-07-22T23:18:36.000Z

One kind-of heavy-weight option, used by ue4-docker, is running a service outside the container that exposes the secret, and have the container image build pull that secret over the network connection.

You can see ue4-docker's implementation of the external host and a Git Credential Helper to query it, including a generated token to provide the bare-minimum protection.

A significant cost to this approach is that the token ends up in the image as a build-arg, so layer caching is confounded; I don't know if the same thing happens when using BuildKit secret mounts, but I assume not.

In ue4-docker's case, the container builds are done under control of the script, so the credential endpoint is served by the script, rather than being left running all the time, which make this approach more feasible than it would be for just a Dockerfile being distributed for use with docker build.

Answer 23 · 2021-08-22T16:01:03.000Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

Answer 24 · 2021-08-24T14:04:37.000Z

I'd love to assist, as I really need this feature to speed up the process of building Docker Windows images.

Is there any way to build stages in parallel in Windows without using BuildKit?

Answer 25 · 2021-09-23T16:01:04.000Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

Answer 26 · 2021-11-03T16:03:34.000Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

Answer 27 · 2021-11-03T18:09:33.000Z

Can somebody please freeze this issue? 😕

Answer 28 · 2021-11-03T19:33:01.000Z

I don't know if we can freeze issues. If anyone knows, let me know :).

I could use help to be enlightened on the benefit or the pain points this can address. I see these so far in this thread:

containerd parity
Run Linux and Windows container side by side
Can use secrets

What else?

How does this help in your or your company's modernization effort? Is this really Developer centric or it can add value to your DevOps in production?

Answer 29 · 2021-11-03T20:55:35.000Z

We would like this so that eventually docker buildx bake support can be added so that we can define one hcl file with all our image builds declaratively configuring things like image tags and args, along with parallel builds out of the box. We use this on linux right now and it works really well.

Answer 30 · 2021-11-05T01:35:14.000Z

What else?

Off the top of my head, I'd bring up:

BuildKit automatically parallelizes multi-stage builds for you. It's probably my favorite feature. See:
https://www.gasparevitta.com/posts/advanced-docker-multistage-parallel-build-buildkit/
If you manage a container build farm or CI system with many workers, BuildKit makes it easy to share Docker build caches across several machines, which can speed up similar builds across them. See:
https://link.medium.com/VYWZJPwnVkb
Other projects exclusively dependant on BuildKit would finally support Windows, like the wonderful "Earthly", sort of Makefile Dockerfile hybrid built upon the BuildKit core APIs. See:
https://earthly.dev
(Only for Linux containers at this time due to the missing Windows BuildKit support.)

If I think of more I'll extend this list. 🤔

Answer 31 · 2021-11-05T19:50:07.000Z

I'd like to add:

passing secrets in as build args

Answer 32 · 2021-11-05T21:37:58.000Z

I remembered another big useful feature: Buildkit has "build-time mounts" support.

Everyone knows about --mount=type=secret to access secret files without baking them in the image layers, but did you know Buildkit also offers:

--mount=type=bind to bindmount non-sensitive stuff at build time temporarily that that is outside of your build context.
--mount=type=cache to make a cache path survive between runs, speeding up your builds. You can use this to cache Go packages, pip build caches, Rust crate caches, apt-get caches and so on!
Mounting secrets is one thing, but if you're just using them for SSH access don't do that and use --mount=type=ssh which lets you do SSH agent forwarding for a singular RUN statement, making SSH connections magically work at build time without having to bake in your keys or mount them via secret mounts.

Answer 33 · 2021-11-06T08:04:37.000Z

Not being able to pass secrets into the Windows container build is painful - for us it means copying down files to the container build host (since that requires a secret) and then having docker slowly discover the container build context and copy the files into the container during the container build (double I/O).

It would be much better to be able to just download the files inside the container as part of the container build.

Answer 34 · 2021-12-06T16:01:14.000Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

Answer 35 · 2022-01-05T16:01:19.000Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

Answer 36 · 2022-02-05T16:01:24.000Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

Answer 37 · 2022-03-08T16:01:23.000Z

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

Answer 38 · 2022-03-08T17:25:10.000Z

Thank you @msftbot for the monthly reminder that nothing has changed. 👀

Answer 39 · 2022-03-08T17:54:25.000Z

What do you expect from Microsoft here? You want them to write and submit patches that add Windows support to BuildKit? Why they would do it? Looking at how microsoft/hcsshim#901 is stuck waiting for approval for around a year, it would be more effective to remove Microsoft from dependency chain of this feature.

Answer 40 · 2022-03-13T05:17:58.000Z

What do you expect from Microsoft here? You want them to write and submit patches that add Windows support to BuildKit? Why they would do it?

@slonopotamus because of the huge differences in how windows containers and linux containers work. I dought that anyone else has enough knowledge about the internals to do it...

Answer 41 · 2022-03-15T21:02:48.000Z

Hi all, in effort to reduce spam of the beloved msftbot asking for status ten times over, I am going to freeze conversation on this issue (hoping it pauses the bot spam in the process). Given this issue has over 80 emoji reactions, we hear you loud and clear this is something you as developers would like to see. If and when we have a concrete update to share on our investments in this area, we will reply here and unfreeze the issue.

Answer 42 · 2022-08-20T16:03:47.000Z

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

Answer 43 · 2022-09-02T12:56:23.000Z

Checking the latest status

Answer 44 · 2022-09-02T15:27:49.000Z

Current status on my end:

microsoft/hcsshim#901 has cleared its legal concerns, but needs some updating and the rest of its tests written. (Which hopefully will reproduce the issue I've seen in containerd with this code, so I can fix it)
containerd/containerd#4419 is the last containerd PR in the series of PRs that should fulfill BuildKit's requirements to function on WCOW. However, it still has a snapshotter test suite failure (on LTSC2019), and a bunch of stuff in there is somewhat hacky and needs to be unhacked.
Then someone can come back to moby/buildkit#616 and work through whatever's bitrotted in BuildKit since I got (hacky, incomplete, and rather faulty) WCOW support working years ago.

I do plan to continue trying to get this working, but I'm not going to be able to do much for the next month or two at least, due to personal-life things, e.g., moving home.

I don't know how BuildKit integrates into docker buildx, so it's possible that some of the above could be shortcut by the upcoming Docker 22.06 release's support for containerd on Windows. However I suspect that at the very least microsoft/hcsshim#901 would need to be completed and integrated into Docker's storage system; possibly it'd also need containerd/containerd#4419 since parts of that are directly compiled into BuildKit.

Answer 45 · 2022-11-03T14:49:51.000Z

I dump again into the deep here. where are we with containerd and building windows containers with podman, nerdctl?
cc: @jsturtevant

Answer 46 · 2022-11-03T17:08:13.000Z

nerdctl uses build kit. Buildkit support is still in progress, you can see the relevant PRs in #34 (comment).

I am not sure how podman builds containers.

Answer 47 · 2022-11-03T21:10:45.000Z

@TBBle any update from your side?
I am checking myself now.

Answer 48 · 2022-11-05T03:16:20.000Z

Nothing from me personally since September 3rd, I'm still on sabbatical. One or two other people have been pushing at the same and related PRs in my absence, but I can't report any measurable progress here.

Answer 49 · 2022-11-08T18:38:42.000Z

Nothing from me personally since September 3rd, I'm still on sabbatical. One or two other people have been pushing at the same and related PRs in my absence, but I can't report any measurable progress here.

Thanks, keep enjoying your ooo.

Answer 50 · 2022-11-15T03:26:06.000Z

@Juarezhm will be on point to drive this from the Microsoft side. @TBBle and @gabriel-samfira appreciate your help here. :)

Answer 51 · 2022-11-15T15:04:52.000Z

I'll specifically tag for visibility/credit @gabriel-samfira who's been kicking the tyres and improving the various PRs I mentioned above (as well as other parts of the WCOW/BuildKit work), as I've just noticed they're not mentinoed on this ticket that I can see. (But may be subscribed already, I can't tell.)

Answer 52 · 2022-11-15T15:40:45.000Z

@TBBle thanks for the tag! A couple of more PRs will be proposed this week to related projects and buildkit itself. The goal is to have buildkit running on windows soon.

Answer 53 · 2022-11-25T07:17:11.000Z

@TBBle thanks for the tag! A couple of more PRs will be proposed this week to related projects and buildkit itself. The goal is to have buildkit running on windows soon.

awesome news!

Answer 54 · 2022-12-25T16:02:49.000Z

This issue has been open for 30 days with no updates.
@Juarezhm, please provide an update or close this issue.

Answer 55 · 2023-02-07T00:18:04.000Z

Just a quick update. All of the different moving parts needed to get this supported, have been proposed in the various projects involved. I'm just ironing out the rough edges in the various PRs and getting everything in a state that can be merged. Quick demo of the WiP build (ran on a linux machine, against a remote buildkitd running on Windows):

Answer 56 · 2023-02-07T00:22:58.000Z

Just a quick update. All of the different moving parts needed to get this supported, have been proposed in the various projects involved. I'm just ironing out the rough edges in the various PRs and getting everything in a state that can be merged. Quick demo of the WiP build (ran on a linux machine, against a remote buildkitd running on Windows):

So exciting!!!

Answer 57 · 2023-02-23T07:39:05.000Z

@gabriel-samfira -- good stuff! looking forward to your update on this. Also let me know if any help is needed to fast-track.

Answer 58 · 2023-02-23T11:15:57.000Z

Hi @profnandaa !

At this point I'm just waiting in the review queue for the PRs already opened in the various projects and following the usual process. Things are progressing well. A list of the current PRs needed to get Windows support enabled (sorted by priority in which they need to merge) is as follows:

Some of these may be split up into more PRs to make things easier to review. Some are marked as draft as their dependencies need to merge first before they can build.

Answer 59 · 2023-03-02T06:20:43.000Z

Hi @profnandaa !

At this point I'm just waiting in the review queue for the PRs already opened in the various projects and following the usual process. Things are progressing well. A list of the current PRs needed to get Windows support enabled (sorted by priority in which they need to merge) is as follows:

microsoft/go-winio#274

microsoft/hcsshim#1637

containerd/containerd#8043

moby/buildkit#3322

moby/buildkit#3517

moby/buildkit#3518

moby/buildkit#3544

moby/buildkit#3545

Some of these may be split up into more PRs to make things easier to review. Some are marked as draft as their dependencies need to merge first before they can build.

Awesome 👍

Answer 60 · 2023-04-21T17:30:46.000Z

This issue has been open for 30 days with no updates.
@Juarezhm, please provide an update or close this issue.

Answer 61 · 2023-04-21T17:30:54.000Z

This issue has been open for 30 days with no updates.
@Juarezhm, please provide an update or close this issue.

Answer 62 · 2023-04-25T21:31:43.000Z

Still in motion...

Answer 63 · 2023-05-17T15:15:55.000Z

To clarify, would this allow us to utilize BuildKit on Linux and build Windows images with the --platform=win/amd64 flag ? Also allow us to utilize remote driver builds with BuildKit daemon? Sorry if I misunderstood the scope.

Answer 64 · 2023-05-17T15:41:28.000Z

Hi @gaui

The scope is to enable buildkitd to run natively on Windows and build Windows container images. You will also be able to use it remotely from any system where you have a compatible client available (ie: buildctl).

Answer 65 · 2023-05-22T16:50:35.000Z

@gabriel-samfira Is there any place to watch for the release other than this issue?

Answer 66 · 2023-05-22T17:05:46.000Z

@gaui Once the PRs opened by me on the buildkit project are reviewed and merge, this support should be in. You can view them here:

https://github.com/moby/buildkit/pulls/gabriel-samfira

Some PRs may be split in smaller PRs.

The current PRs are in various stages of review. Some are draft as dependencies need to merge first, some need to be rebased but that will only happen once their dependencies merge, some are awaiting further reviews. A number of them have merged already in buildkit and the various other projects that buildkit depends on. Most notably, in containerd.

Currently waiting for reviews.

Answer 67 · 2023-05-23T13:50:28.000Z

I suspect for a lot of users, we can't fully resolve this issue until platform support flows through into buildx and any related Dockerd plumbing, given that the original issue was quoting from the Docker documentation. Although that should hopefully be easy-to-deliver plumbing compared to what's already been landed, and particularly, easier to do end-to-end iteration and wider testing with, compared to the lower-level components.

Answer 68 · 2023-05-23T14:15:41.000Z

I suspect for a lot of users, we can't fully resolve this issue until platform support flows through into buildx and any related Dockerd plumbing, given that the original issue was quoting from the Docker documentation. Although that should hopefully be easy-to-deliver plumbing compared to what's already been landed, and particularly, easier to do end-to-end iteration and wider testing with, compared to the lower-level components.

Yes. Once we get support (tests included) into buildkit, enabling it in the various frontends/plumbing should be a lot easier. Even without enablement in buildx or docker, it could still be used via buildctl until we merge the needed PRs in the rest of the projects.

The end goal is to have proper enablement in all the current projects.

Answer 69 · 2023-05-23T16:59:15.000Z

Do you have any timelines you can share @gabriel-samfira?

Answer 70 · 2023-05-23T17:29:15.000Z

@fady-azmy-msft Sadly, no. It depends on the availability the buildkit folks have to review the PRs. Right now I believe they are busy finalizing the next release of buildx. I am hoping that after that happens, they can spare some more time to review the above mentioned PRs.

One of the (larger) PRs is slated for the 0.12 release of buildkit. So, fingers crossed.

Answer 71 · 2023-06-08T14:12:05.000Z

A short update (to mostly keep @microsoft-github-policy-service happy):

The needed containerd functionality was backported to the stable 1.7 branch and released. The updated dependency was merged in buildkit as well (moby/buildkit#3782)
The above merge unblocked moby/buildkit#3783
The moby/buildkit#3322 PR was split in 3:
Two of the three were merged

As a side note, once moby/buildkit#3783 and moby/buildkit#3908 merge, a very basic Dockerfile can be built on Windows (example here: moby/buildkit#3783 (comment)). Any Dockerfile that does not involve the executor (RUN) or determining if a user exists (WORKDIR, USER, ADD --chown=, etc).

I am updating the executor PRs and will push them as soon as their dependencies merge (moby/buildkit#3783 and moby/buildkit#3908).

Answer 72 · 2023-07-10T09:19:52.000Z

This issue has been open for 30 days with no updates.
@Juarezhm, please provide an update or close this issue.

Answer 73 · 2023-07-10T09:19:58.000Z

This issue has been open for 30 days with no updates.
@Juarezhm, please provide an update or close this issue.

Answer 74 · 2023-07-10T21:09:40.000Z

It does look like all the PR's has been merged, so I don't see a reason for this to be closed (even though the bot wants to) 🙂

Answer 75 · 2023-07-11T05:19:11.000Z

It does look like all the PR's has been merged, so I don't see a reason for this to be closed (even though the bot wants to) 🙂

There are a few more that need to merge. Namely the executor branch and the branch implementing readUser() described here: #34 (comment)

Both these branches need to be rebased and possibly split into multiple smaller branches. What has merged yesterday, was the file path handling PR which normalizes and converts paths based on OS. Buildkit should no longer fail to parse paths that contain drive letters, will be able to handle paths in dockerfiles that contain backslashes and will properly normalize the WORKDIR on Windows.

The executor is needed to actually build useful images.

Answer 76 · 2023-08-10T15:29:09.000Z

This issue has been open for 30 days with no updates.
@Juarezhm, please provide an update or close this issue.

Answer 77 · 2023-08-10T15:29:13.000Z

This issue has been open for 30 days with no updates.
@Juarezhm, please provide an update or close this issue.

Answer 78 · 2023-08-10T15:47:39.000Z

Does it close the ticket, if nothing happens?

Answer 79 · 2023-08-10T15:50:38.000Z

No, it just spams you until you give up any hope and unsubscribe.

Answer 80 · 2023-08-10T16:00:34.000Z

Work on this was paused for a bit, but has now resumed. I am rebasing the existing 2 PRs now:

With those merged, we should have the ability to build useful Dockerfiles. After that we'll need to enable integration tests on Windows to ensure nothing breaks in the future.

Sorry for the delay on this.

Answer 81 · 2023-08-23T17:02:19.000Z

@iankingori

Answer 82 · 2023-08-30T21:18:50.000Z

Quick update (mostly to keep @microsoft-github-policy-service happy) : One of the last two PRs in this series has merged. Only one remains.

Answer 83 · 2023-08-31T13:28:05.000Z

If anyone is feeling brave enough, I wrote a gist with install instructions here: https://gist.github.com/gabriel-samfira/44092adae911fa625c33c829ac39d736

This will only run on ltsc2019 and ltsc2022. You will need a fresh install.

Note, until the last branch merges, any operation that requires user resolution, will fail. This means WORKDIR, ADD (with the --chown flag set - works without it).

Edit: NOTE: This will install the master branch of buildkit.

Answer 84 · 2023-09-01T09:21:41.000Z

@gabriel-samfira Will this enable us to run buildkitd in a container with privileged access and use that container to build other Windows images?

Answer 85 · 2023-09-01T09:33:42.000Z

Hi @alexproca !

In theory, it should work in a host process container. I have not tested that scenario yet, though. The goals in order are:

get something useful to work (we are here)
enable integration tests and augment existing CI pipelines to run tests on Windows
make the necessary ecosystem changes to take advantage of the new functionality (buildx, nerdctl, etc)
smooth out rough edges across the board (enable missing functionality like mounting SSH sockets, secrets, etc)
packaging

Of course, anyone can attempt to run it in a host process container and package it if they wish. In theory, as long as it can access the containerd named pipe (which a host process container should be able to do), it will work.

Answer 86 · 2023-10-04T16:59:05.000Z

@lucillex is taking over from @Juarezhm.

Answer 87 · 2023-10-23T07:57:29.000Z

The final branch in this series has merged. The master branch of buildkit can now build windows images, using Windows Server SKUs.

Answer 88 · 2023-10-26T06:35:19.000Z

Side question about this huge amout of work.
Does it means it's now possible to build a windows image from a linux OS as an example ?
Is it possible to build a windows ltsc2022 by running on windows 10 or server 2019 ?

Answer 89 · 2023-10-26T07:47:11.000Z

No. You can only build Windows containers on Windows.

Answer 90 · 2023-10-26T08:11:55.000Z

But with Buildkit becoming available for Windows it should be possible to build LTSC2022 on LTSC2019/Windows 10, when using a remote builder in a LTSC2022 HyperV-VM, right?

Answer 91 · 2023-10-26T08:48:51.000Z

Does it means it's now possible to build a windows image from a linux OS as an example ?

You will be able to set up a remote buildkitd daemon and connect to it from a linux machine. This demo of a WiP was run from a linux machine: #34 (comment)

Is it possible to build a windows ltsc2022 by running on windows 10 or server 2019 ?

I believe you should be able to build Windows Server 2019 images on a ltsc2022, but not the other way around. I have not tested this scenario yet. Work was focused on getting something working at all. I will have to check. There have been efforts to make Windows containers a bit more forward compatible (run newer versions of Windows containers on older hosts), but that work started with ltsc2022. @fady-azmy-msft correct me if I'm wrong. There is also some platform matcher work in containerd meant to address some of the rough edges (CC: @aznashwan)

Answer 92 · 2023-10-26T10:59:05.000Z

Great ! Thank you for that answers.

So we still need a "runner" in the same OS (which makes sense), but at least we can make a remote connection, and eventualy host that runner on HyperV or VirtualBox with a minimal installation of windows.

For the versions compatibility I would be curious about what you find. The fact to build a version >= from a specific windows version can be more or less complex, expecialy when a new major version is published but you don't necessarly want / have time to upgrade all your runners first.
But I guess there is a technical reason behind...

Answer 93 · 2023-10-26T12:11:43.000Z

So we still need a "runner" in the same OS (which makes sense), but at least we can make a remote connection, and eventualy host that runner on HyperV or VirtualBox with a minimal installation of windows.

Would be fantastic if you could share your knowledge here and/or code if you planing to test this out. This could make building images far easier on local machines.

Answer 94 · 2023-10-26T12:18:43.000Z

If you're getting the error: failed to solve: failed to load cache key: no match for platform in manifest: not found message with the latest master branch, that is most likely from the containerd platform matcher mentioned earlier.

We'll have a look at it as soon as possible. In the meantime, an ugly hack is to use the hash of the image from within the manifest:

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:2bd9f83d70574aa718364abce9973b364347460ed21ac00a31e42e3b3c815824

You can inspect the manifest with:

docker manifest inspect mcr.microsoft.com/windows/nanoserver:ltsc2022
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 429,
            "digest": "sha256:2bd9f83d70574aa718364abce9973b364347460ed21ac00a31e42e3b3c815824",
            "platform": {
                "architecture": "amd64",
                "os": "windows",
                "os.version": "10.0.20348.2031"
            }
        }
    ]
}

Answer 95 · 2023-10-26T12:37:05.000Z

@gabriel-samfira -- another option is to specify the arch tag (-amd64) for the exact manifest:

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022-amd64

I'm currently testing the new changes on Server2019.

So far what I've determined, the above issue not withstanding:

You can build ltsc2022 images on Windows 11 (and Server 2022)
You can't build down-level images, i.e. ltsc2019 on Windows 11 and Server 2022, you end up with ERROR: no match for platform in manifest: not found

Answer 96 · 2023-10-26T12:43:23.000Z

https://learn.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility?tabs=windows-server-2022%2Cwindows-11
you can use process isolation only when OS version and container version matching (Windows 11 supports Windows Server 2022 images), you can use hyper-v isolation (which have limitations, not everything what working in process isolation working there) to run images of older versions on newer OS, you cannot run newer images on older OS

Answer 97 · 2023-10-26T13:43:01.000Z

If you want to try out HyperV Isolation, BuildKit master branch (as of moby/buildkit#4364) can pass through the necessary options from its own buildkit.toml config file. I haven't tested this, but it may be as simple as passing the SandboxIsolation=1 flag, similar to what containerd/cri does:

[worker.containerd.runtime.options]
  SandboxIsolation = "1"
  ScaleCpuLimitsToSandbox = true

Assuming I have my TOML syntax correct here.

Note that you'll definitely need to pick a non-index FROM, as even in containerd, platform matcher specialisation for HyperV Isolation is not a solved issue.

For the record, this is also how one changes the debug level of the shim, or customises things like the number of CPUs or amount of RAM in the HyperV isolation case. The latter is something that was never hooked up in Docker, so chalk that up as another win for BuildKit. ^_^ (There's a bunch more possible adjustable flags available by container and VM annotations, but I don't immediately see a way to pass them through from BuildKit. Runtime options support was itself only added to BuildKit in September, and so adding annotation support should be reasonable doable.)

As far as Windows 11 hosts go, until we get Stable ABI support in the BuildKit platform matcher, when you look at a Windows base image tag list on either MCR or DockerHub, it indicates which ones are manifest lists ("multiarch" on DockerHub, even though they mostly contain only one arch right now) and which are just images. For now, appending -amd64 to the manifest lists seems like a pretty consistent way to get the specific image.

Answer 98 · 2023-11-14T16:33:50.000Z

Thank you very much for the great work you are doing.
Is it possible to run buildkit in a container on Kubernetes (Windows Nodes) to build Windows images?

We use this setup for linux containers:

image: moby/buildkit:v0.12.3-rootless
     volumeMounts:
     - name: work
       mountPath: /work
     - name: docker-config
       mountPath: /.docker
     command:
     - buildctl-daemonless.sh
     args:
       - build
       - --frontend
       - dockerfile.v0
       - --local
       - context=.
       - --local
       - dockerfile=./{{inputs.parameters.path}}
       - --output
       - type=image,name=myacr.azurecr.io/{{inputs.parameters.image}}:{{inputs.parameters.tag}},push=true
     env:
       - name: BUILDKITD_FLAGS
         value: --oci-worker-no-process-sandbox
       - name: DOCKER_CONFIG
         value: /.docker

Answer 99 · 2023-11-14T22:52:43.000Z

I don't think anyone's tried putting BuildKit for Windows in a container. If it is possible, it'll almost certainly require a host-process container (the Windows equivalent of 'privileged') as I don't think we can support "rootless" behaviour on Windows.

Our current intent is to get BuildKit's integration tests working uncontainerised first.

(Also, the platform matching issue has been resolved, so the only major issue I'm tracking now is the PATH being wrong on non-nanoserver base images.)