Starting a container with memory limit below 8M leaves zombie system processes and unstoppable container

[smallmodel]

Describe the bug
When starting a container in process isolation mode, with memory limit below 8M, it hangs and cannot be stopped.

To Reproduce

docker run --isolation process --rm -ti -m 6M mcr.microsoft.com/windows/servercore:ltsc2022

Expected behavior
The container must terminate if the Windows environment doesn't have enough memory to initialize.

Configuration:

• Edition: Windows 11
• Base Image being used: Windows Server Core
• Container engine: docker
• Container Engine version: 25

Additional context
This may be due to fontdrvhost.exe not starting, system processes are stuck waiting in kernel-mode for umfd initialization.

[ntrappe-msft]

🔖 ADO 50069941. Will try to repro but may hand off to another team member.

[ntrappe-msft]

Thank you for bringing up this Issue! It's made us aware of areas of improvement around limited memory situations in containers and how we want to manage that. I'm assuming you selected 6M (6 megabytes) as the memory limit because the Docker docs state that as the minimum.

Unfortunately, that's only for Linux containers. Windows containers require more memory, usually in the hundreds of megabytes to start cleanly. We'll continue to work on this Issue, but, as a rule of thumb, we'd recommend the following:

• For Nano Server images, use 80MB minimum
• For Server Core images, use 200MB minimum

[smallmodel]

In the runtime options, there is the --kernel-memory parameter, it is accepted but doesn't seem to do anything with a low value (6M): for example, an unlimited number of kernel objects could be created.

As an example, an infinite number of window stations (I though it would be a good test as win32k is a per-session driver) can be created without limitations. Here is a test code:

#include <Windows.h>
#include <tchar.h>
#include <string>
#include <iostream>

static const std::wstring PdevPrefix = L"OVERFLOW_";
static const std::wstring DefaultWindowStationName = L"WS_";

int main(int argc, char* argv[]) {
    std::wstring WindowStationName;

    unsigned long i;
    HWINSTA hWindowStation;
    USEROBJECTFLAGS uof;

    uof.fInherit = FALSE;
    uof.fReserved = FALSE;
    uof.dwFlags = WSF_VISIBLE;

    for (i = 0;; i++) {
        WindowStationName = PdevPrefix + DefaultWindowStationName + std::to_wstring(i);

        hWindowStation = CreateWindowStation(WindowStationName.c_str(), 0, GENERIC_ALL, NULL);
        if (!hWindowStation) {
            std::cout << "Window station creation failed: " << GetLastError() << std::endl;
            continue;
        }

        SetUserObjectInformation(hWindowStation, UOI_FLAGS, (PVOID)&uof, sizeof(uof));
    }
}

It was able to use more than 2GB of paged memory with --kernel-memory 6M.

[ntrappe-msft]

@smallmodel Hi, I believe the --kernel-memory parameter (used to limit kernel memory for a container) is a Linux-specific feature. It does not apply to Windows Containers. If you'd like to limit the memory (commit-memory) used by Windows Containers, you'll have to use the -m flag. I'm closing this Issue for now as containers should work fine with respect to the minimum memory we've recommended. Please open it again if you encounter more problems that haven't been addressed.