Internal Host DNS cannot be resolved (Windows Container 1803)

Question

Internal Host DNS cannot be resolved (Windows Container 1803)

n-junge opened this issue 8 years ago · 109 comments

Expected behavior

According to the Docs host.docker.internal resolves the host's ip.

Actual behavior

Ping request could not find host host.docker.internal
Network works fine. Pinging Host IP directly (ipconfig) works as expected.

Host: Windows 10 Enterprise
Container: microsoft/windowsservercore:latest

Neither stable (18.03) nor edge (18.04) works.

Probably important side note: DNS can be resolved in Linux Container (Ubuntu)

 PS C:\> ping host.docker.internal
 Ping request could not find host host.docker.internal. Please check the name and try again.
 PS C:\> ping gateway.docker.internal
 Ping request could not find host gateway.docker.internal. Please check the name and try again.
 PS C:\> ping google.com
 
 Pinging google.com [172.217.16.78] with 32 bytes of data:
 Reply from 172.217.16.78: bytes=32 time=31ms TTL=46
 Reply from 172.217.16.78: bytes=32 time=31ms TTL=46
 Reply from 172.217.16.78: bytes=32 time=31ms TTL=46
 Reply from 172.217.16.78: bytes=32 time=31ms TTL=46
 
 Ping statistics for 172.217.16.78:
     Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
 Approximate round trip times in milli-seconds:
     Minimum = 31ms, Maximum = 31ms, Average = 31ms

C:\WINDOWS\system32>docker version
Client:
 Version:       18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:06:28 2018
 OS/Arch:       windows/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:      18.03.0-ce
  API version:  1.37 (minimum version 1.24)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:        Wed Mar 21 23:21:06 2018
  OS/Arch:      windows/amd64
  Experimental: false

Answer 1 · 2018-04-26T19:06:09.000Z

Hi @4xiz
Thanks for reporting - I can repro and have reported it as a bug

Answer 2 · 2018-05-02T08:55:26.000Z

It's a windows daemon limitation and needs to be fix there but we will add a workaround waiting MS to fix it upstream.

Answer 3 · 2018-05-17T15:11:32.000Z

Release 18.05.0-ce-win66 works. Thanks.

Answer 4 · 2018-05-17T19:47:23.000Z

Tested it on Windows 10, Docker 18.05.0-ce-win66 (17760) and it doesn't work

Answer 5 · 2018-05-18T09:27:03.000Z

@atomaras
Well the dns resolution seems to work for you too. So that's a different problem. I propose you open another issue.

Answer 6 · 2018-06-20T22:07:58.000Z

I just tried this and it isn't working for me. Version info:

Windows
Microsoft Windows [Version 10.0.17134.112]

Docker

Client:
 Version:      18.05.0-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   f150324
 Built:        Wed May  9 22:12:05 2018
 OS/Arch:      windows/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.05.0-ce
  API version:  1.37 (minimum version 1.24)
  Go version:   go1.10.1
  Git commit:   f150324
  Built:        Wed May  9 22:29:00 2018
  OS/Arch:      windows/amd64
  Experimental: false

Ping output:

C:\>docker exec b61746027bc9 ping host.docker.internal
Ping request could not find host host.docker.internal. Please check the name and try again.

Answer 7 · 2018-06-21T00:19:05.000Z

@JoshCollinsMSFT I have the same windows build (17134.112) and docker (18.05.0-ce) and am also not able to resolve host.docker.internal. No joy with connection strings.

Answer 8 · 2018-08-14T07:05:46.000Z

Still not working for me with 18.06.0-ce:

docker run --rm -i microsoft/nanoserver:1709 ping host.docker.internal
Ping request could not find host host.docker.internal. Please check the name and try again.

Answer 9 · 2018-08-19T22:50:57.000Z

Same problem. docker.host.internal not working.

Answer 10 · 2018-08-20T05:26:10.000Z

@n-junge @ebriney could you please re-open this issue?

Answer 11 · 2018-08-23T13:52:06.000Z

So it seems the way this was fixed is that it must write out to the local hosts file on the host machine.

I have a bit of a situation as Symantec Endpoint blocks write access to the hosts file :(.
Is there a better dns way of doing this?
Thanks

Answer 12 · 2018-08-24T14:08:12.000Z

They are 2 separate things.

First, for dockerd to get host.docker.internal resolution we must put it in windows hosts file...
So if you have endpoint protection or similar you must add an exception on com.docker.service executable.

Second, in the windows containers, we do a docker exec as soon as the container is started (using docker event) to patch the hosts file in the container (I don't know what happened if you are running in process isolation, because we don't support Windows server). So if you overwrite it at runtime you can lose those entries.

@Heurazio this is host.docker.internal, not docker.host.internal. you also have gateway.docker.internal but that is the same ip in windows containers.

@prochnowc can you try docker run microsoft/nanoserver powershell "Start-Sleep -s 2 ; if (! $(IPCONFIG /displayDNS | Select-String -Quiet -SimpleMatch host.docker.internal)) { exit 1 }"

Answer 13 · 2018-08-24T14:21:18.000Z

@ebriney Interestingly ... if i wait 2 seconds after startup i can ping host.docker.internal. I will try this again within my traefik for windows container. traefik needs the gateway address.

Answer 14 · 2018-08-24T14:31:24.000Z

Unfortunately it does not work.. traefik_1 | time="2018-08-25T01:30:05+02:00" level=error msg="Failed to retrieve information of the docker client and server host: error during connect: Get http://host.docker.internal:2375/v1.24/version: dial tcp: lookup host.docker.internal: getaddrinfow: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server."

Does the host.docker.internal stuff require minimum nanoserver version?

Answer 15 · 2018-08-27T08:22:36.000Z

@prochnowc, we execute that script as soon as we receive the start event from any windows containers:

$hostsFile = "C:\Windows\System32\drivers\etc\hosts"
$src = [System.IO.File]::ReadAllLines($hostsFile)
$a = $src += ""
For ($i=0; $i -le $a.length; $i++) {
	if ($a[$i].Contains("host.docker.internal"))
	{
		$a[$i] = "%s host.docker.internal"
		$a[$i+1] = "%s gateway.docker.internal"
		[System.IO.File]::WriteAllLines($hostsFile, [string[]]$a)
		exit
	}
}
$a = $a += "# Added by Docker for Windows"
$a = $a += "%s host.docker.internal"
$a = $a += "%s gateway.docker.internal"
$a = $a += "# End of section"
[System.IO.File]::WriteAllLines($hostsFile, [string[]]$a)

Perhaps the hosts file doesn't exist and there is an exception reading it.

Answer 16 · 2018-08-27T08:38:39.000Z

@ebriney Am I right when saying that this requires working PowerShell in the windows container? My traefik container does not have PowerShell.

Answer 17 · 2018-08-27T08:44:00.000Z

@prochnowc Arfff, that's why it didn't work. Do you have a suggestion on how to solve that?

Answer 18 · 2018-08-27T15:18:41.000Z

@ebriney So actually this fix only works on windows containers which have PowerShell installed (which is not the case for windows nanoserver). I dont think the hosts file editing can be done with "standard" windows command line utils. My idea would be to implement this in the docker DNS.

Answer 19 · 2018-08-27T15:26:31.000Z

@prochnowc hummm, nanoserver includes powershell (see command line above).
And yes your image must include powershell for now.
Microsoft is managing all the network part on WCOW so except if they handle it upstream we cannot do anything (but to do that I think it must be generalized in moby/moby).
On Linux containers the vm traffic is going throught vpnkit so we can tweak dns requests ...

Answer 20 · 2018-08-29T08:31:11.000Z

@ebriney Sorry, but nanoserver does not include PowerShell by default. See https://docs.microsoft.com/en-us/windows-server/get-started/nano-in-semi-annual-channel. Your command line didnt work with pure nanoserver image. I had to use mcr.microsoft.com/powershell:6.0.2-nanoserver.

Answer 21 · 2018-08-29T08:54:00.000Z

and so why docker run microsoft/nanoserver powershell "Start-Sleep -s 2 ; if (! $(IPCONFIG /displayDNS | Select-String -Quiet -SimpleMatch host.docker.internal)) { exit 1 }" is working?

Answer 22 · 2018-08-29T08:54:54.000Z

ok that's in 1803 image

Answer 23 · 2018-09-03T15:56:13.000Z

@ebriney , Would you be able to update your powershell script slightly to only write to the host file if needed?

I can see you do an if string contains "host.docker.internal" but then also add the comment strings to the file after the check by the looks of it.

Because of this second write to the file outside of this condition, this one is causing me a bit of pain because I've had my I.T update my hosts file manually to fix this, but because of this second write statement that happens unconditionally, when I click switch to Windows containers, there is a "can't write to hosts file error" still and crashes Docker.

Also in my case the wrong host ip was getting selected too as we have multiple network adaptors.

A suggestion if you could make if possible, would be to first check the DNS resolves first, if it doesn't then check if the hosts file has the additions it needs, and write to it if its not in there, also would move the second write in to the if statement.

Hope this makes sense.

I've coded and tested my suggestions above (W10) in the below, so feel free to use parts or all of it if needed:
Really appreciate any help as ping ponging with my IT dept here on this which is painful.

$hostsFile = "${env:windir}\system32\drivers\etc\hosts"
# Check DNS resolves before touching the hosts file.
if((Resolve-DnsName -Name host.docker.internal) -And (Resolve-DnsName -Name gateway.docker.internal)){}
else {
    # Check if in the hosts file and write it in if its not there...
    if((cat $hostsFile | Select-String -Pattern "host.docker.internal") -And (cat $hostsFile | Select-String -Pattern "gateway.docker.internal")){}
    else {
        $ip = (Resolve-DNSName (Hostname) | Select IPAddress | select IPAddress -expandproperty IPAddress | select -last 1)
        $addtohost = "`n#Added by Docker for Windows`n$ip    host.docker.internal`n$ip    gateway.docker.internal`n# End of section`n"
        echo $addtohost | Out-File -Append -filepath $hostsFile
    }
}

Answer 24 · 2018-09-05T08:24:27.000Z

Ok @kierzo, thanks for the feedback, I'll fix it.
And will try to re-write it in batch when I will have some time so it can work in containers without powershell (cc @prochnowc).

Answer 25 · 2018-09-05T08:40:43.000Z

@ebriney Excellent! thanks a lot :)

Answer 26 · 2018-09-05T09:36:18.000Z

@kierzo when you do
$ip = (Resolve-DNSName (Hostname) | Select IPAddress | select IPAddress -expandproperty IPAddress | select -last 1) is it in the container or in the host?

Answer 27 · 2018-09-05T11:31:16.000Z

@ebriney , on my host, that's where I have the problem. haven't tried in a container.

I did attempt a rewrite in batch if it helps, works on my W10 test, it may help you possibly?

@echo off
SETLOCAL

SET hostsfile=%systemroot%\system32\drivers\etc\hosts

:: Get local ip
FOR /F "tokens=* USEBACKQ" %%F IN (`hostname`) DO (
	SET hostname=%%F
)
FOR /F "tokens=2 USEBACKQ delims=:" %%F IN (`nslookup %hostname%`) DO (
	SET ip=%%F
)

:: Check if DNS resolves...
ping host.docker.internal -n 1 > NUL 2>&1 && ping gateway.docker.internal -n 1 > NUL 2>&1
IF /I "%ERRORLEVEL%" NEQ "0" (
    ECHO DNS Resolution failed
) ELSE (
	:: If it resolves jump over the step to write to the hosts file...
	goto END 
)

:: Check if in hosts file or not and stick it in if its not....
TYPE "%hostsfile%" | FIND "host.docker.internal" && TYPE "%hostsfile%" | FIND "gateway.docker.internal" 
IF /I "%ERRORLEVEL%" NEQ "0" (
	ECHO. >> %hostsfile%
	ECHO %ip:~2%    host.docker.internal >> %hostsfile%
	ECHO %ip:~2%    gateway.docker.internal >> %hostsfile%
)

:END

:: Check if DNS resolves...
PING host.docker.internal -n 1 > NUL 2>&1 && PING gateway.docker.internal -n 1 > NUL 2>&1
IF /I "%ERRORLEVEL%" NEQ "0" (
    ECHO DNS still failed...
	exit /B 1
)

ENDLOCAL

Answer 28 · 2018-09-05T12:15:39.000Z

Thanks @kierzo, I will use your ip resolution for the host injection and the batch for the container host setup

Answer 29 · 2018-09-19T06:28:39.000Z

@ebriney Should the move from PS to BAT script be in 2.0.0.0-beta1-win75? The ticket was mentioned in the release notes.
Unfortunately it still does not work and I can see
[08:19:57.063][WindowsDaemon ][Info ] debug: exec commandLine: powershell -Command "$hostsFile = \"C:... in the log file.

Answer 30 · 2018-09-19T07:57:44.000Z

@prochnowc : I only did the host part. In the container, I cannot write in hosts file, got an access denied. I need to investigate why (tested on nanoserver:1803 image).

Answer 31 · 2018-09-19T09:27:32.000Z

@ebriney: In nanoserver:1803 the user is ContainerUser which does not have write access. Only ContainerAdministrator has privileged access.

Answer 32 · 2018-09-19T14:57:42.000Z

Hummm, ok I'll try with a -u flag but is it compatible with base nanoserver image (14586) @prochnowc ?

Answer 33 · 2018-09-19T17:35:00.000Z

I dont think that -u flag is supported

Answer 34 · 2018-09-21T05:48:05.000Z

shall we reopen this issue? this issue also happens with microsoft/dotnet:2.1-aspnetcore-runtime-nanoserver-1803 or microsoft/dotnet:2.1-sdk-nanoserver-1803
which is commonly used with asp.net core Dockerfile template.

Answer 35 · 2018-09-26T08:19:11.000Z

I change the title and re-open it @chris-sung.
I found a way to make it works on 1803 using -u ContainerAdmin flag and batch script but unfortunately the same exec command freezes the daemon on old nanoserver images...
I have to relaunch it which is not good at all.

Answer 36 · 2018-09-27T20:58:28.000Z

The issue I'm seeing related to this is that it seems to be getting the IP address from the wrong network.

On container:

C:\>type windows\system32\drivers\etc\hosts
# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host

# localhost name resolution is handled within DNS itself.
#       127.0.0.1       localhost
#       ::1             localhost

# Added by Docker for Windows
172.26.136.33 host.docker.internal
172.26.136.33 gateway.docker.internal
# End of section

C:\>ipconfig

Windows IP Configuration


Ethernet adapter Ethernet 2:

   Connection-specific DNS Suffix  . : *****
   Link-local IPv6 Address . . . . . : fe80::f1f6:737f:2446:d4e0%5
   IPv4 Address. . . . . . . . . . . : 172.21.206.197
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . : 172.21.192.1

On host:

Ethernet adapter vEthernet (nat):

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :

Ethernet adapter vEthernet (Primary Network Card):

   Connection-specific DNS Suffix  . : *****.com
   Link-local IPv6 Address . . . . . : fe80::f1e7:b112:a0fd:49dd%2
   IPv4 Address. . . . . . . . . . . : 10.3.4.220
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 10.3.4.1

Wireless LAN adapter Wi-Fi:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . : *****.COM

Wireless LAN adapter Local Area Connection* 2:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :

Ethernet adapter Bluetooth Network Connection:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :

Ethernet adapter vEthernet (Default Switch):

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::c4b6:1923:2680:2caa%9
   IPv4 Address. . . . . . . . . . . : 172.26.136.33
   Subnet Mask . . . . . . . . . . . : 255.255.255.240
   Default Gateway . . . . . . . . . :

Ethernet adapter vEthernet (nat) 2:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::b2:c004:744a:a107%4
   IPv4 Address. . . . . . . . . . . : 172.21.192.1
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . :

Ethernet adapter vEthernet (c4dc5791e7e9eaa):

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::34ec:7c0c:ecf9:30bb%25
   IPv4 Address. . . . . . . . . . . : 172.18.208.1
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . .
``` . . . . . . :

Even though the container is using nat, it's not using the IP address of the host on that Ethernet card, it's using the primary. I can ping the IP address defined on the NAT network from the container, and it matches the default gateway, which it should.

Answer 37 · 2018-10-08T15:40:10.000Z

Hi @richardgavel, which version are you using?

Answer 38 · 2018-10-09T03:15:54.000Z

Hi @ebriney,

I can confirm this happens as well for Docker EE- version 17.06-2-ee-16

C:\Users\Administrator\Documents>docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 53
Server Version: 17.06.2-ee-16
Storage Driver: windowsfilter
 Windows:
Execution Driver: <not supported>
Logging Driver: json-file
Plugins:
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
Kernel Version: 10.0 14393 (14393.2485.amd64fre.rs1_release.180827-1809)
Operating System: Windows Server 2016 Datacenter
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 16 GiB

May I know if the EE version also having the same code base as the CE version?

Answer 39 · 2018-10-11T14:33:57.000Z

The resolution I could think of was doing an nslookup on your computers host name e.g.

hostname
nslookup myhostname

I'm wondering if this give the correct ip @richardgavel @randytan ?

Answer 40 · 2018-10-15T13:32:08.000Z

Yes I did nslookup the hostname manually and edited the /etc/hosts inside the %Windows%/System32/Drivers/etc/hosts to include two lines of internal host addresses.

These interim solution should do the trick, but for the future releases, it should be working out of the box and the bug should be addressed by either Microsoft or Docker platform.

Answer 41 · 2018-10-15T14:55:57.000Z

@randytan docker ee doesn't have host.docker.internal it's a desktop feature.
and is
(gwmi win32_networkadapterconfiguration | where {$_.ipaddress -ne $null -and $_.defaultipgateway -ne $null} | select -First 1).IPAddress[0]
gives you the right address @randytan @richardgavel @prochnowc ?

Answer 42 · 2018-10-15T15:23:12.000Z

even better perhaps:
(gwmi win32_networkadapterconfiguration | where {$_.ipaddress -ne $null -and $_.defaultipgateway -ne $null} | Sort-Object -Property @{ Expression = "IPConnectionMetric" ; Descending = $True } | select -First 1).IPAddress[0]

Answer 43 · 2018-10-16T09:21:14.000Z

I have a hybrid swarm with both Windows and Linux containers.
From Iinux containers I can do

/ # nslookup db
nslookup: can't resolve '(null)': Name does not resolve

Name:      db
Address 1: 10.0.12.5 2c718448b234.testi_proxy-net

But from Windows ones, I cannot

PS C:\> nslookup.exe db
DNS request timed out.
    timeout was 2 seconds.
Server:  UnKnown
Address:  10.0.13.1

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
*** Request to UnKnown timed-out

My db container runs Windows
Is this issue related? All my Windows containers are 1803 now (I made an upgrade recently).
The Windows node runs v18.03.1-ee-3 and linux ones v18.06.1-ce. Both are latest.

Answer 44 · 2018-10-16T14:37:40.000Z

Hi @achrjulien , this is a separate issue, you should create a new one.

Answer 45 · 2018-11-29T12:12:02.000Z

In case someone comes here and takes the same dodgy path.
I got my problems solved by running 18.06.1-ce on both Linux (Ubuntu/Debian) and Windows (server 1803) on physical servers. For Windows, you have to compile 18.06.1-ce yourself.

Answer 46 · 2019-01-25T10:05:50.000Z

I'm now getting this issue on 18.09.1, I can ping the internet but host.docker.internal does not resolve. This was working before the latest update? I have tried uninstalling both hyperv and Docker but fresh install does not resolve this...

Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 17
Server Version: 18.09.1
Storage Driver: windowsfilter (windows) lcow (linux)
Windows:
LCOW:
Logging Driver: json-file
Plugins:
Volume: local
Network: ics l2bridge l2tunnel nat null overlay transparent
Log: awslogs etwlogs fluentd gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 17134 (17134.1.amd64fre.rs4_release.180410-1804)
Operating System: Windows 10 Pro Version 1803 (OS Build 17134.523)
OSType: windows
Architecture: x86_64
CPUs: 8
Total Memory: 15.86GiB

Update:
downgrading did not resolve this, I have tried adding DNS exceptions into the firewall and disabling, also tried
RUN Set-ItemProperty -path 'HKLM:\SYSTEM\CurrentControlSet\Services\Dnscache\Parameters' -Name ServerPriorityTimeLimit -Value 0 -Type DWord ;
New-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Dnscache\Parameters' -Name MaxCacheTtl -Value 0 -Type DWord ;
New-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Dnscache\Parameters' -Name MaxNegativeCacheTtl -Value 0 -Type DWord ;
Restart-Service dnscache

but that does not work either

nslookup, seems to reveal that it is querying the internal dns on the container nat network but gives no results, querying google works correctly...

Answer 47 · 2019-01-26T17:54:40.000Z

looking at this for a little longer, it seems that the only way i can get the containers to see each other using compose is by defining a hostname. Any aliases, links or service names are ignored on docker-compose up will not resolve but adding hostname: somename then allows the containers to resolve

Answer 48 · 2019-01-28T15:13:20.000Z

Hi @perithompson ,
I'm facing the same problem as you do, but using your hostname workaround leads my services to resolve using ipv6 only. Have you managed getting it work using ipv4?
Unfortunatly there is not option in docker for windows to disable ipv6 in general - as far as I'm aware.

Answer 49 · 2019-01-28T22:17:02.000Z

Hi @perithompson ,
I'm facing the same problem as you do, but using your hostname workaround leads my services to resolve using ipv6 only. Have you managed getting it work using ipv4?
Unfortunatly there is not option in docker for windows to disable ipv6 in general - as far as I'm aware.

Hi @noraab, Yeah, it resolves on ipv6 for me too, but that hasn't caused me a problem for what I needed, I don't know if there is a way to disable it, unless you can change the settings on the hosts virtual switch?

Answer 50 · 2019-01-29T16:00:59.000Z

To provide more details:
This happens on Docker Version 18.09.1
This does not happen on Docker Version 18.09.0

Answer 51 · 2019-02-01T12:19:18.000Z

To provide more details:
This happens on Docker Version 18.09.1
This does not happen on Docker Version 18.09.0

And some more findings:

This does not happen if you use Docker's default network.
This does happen, if you configure your own network within compose-file

I just defined my own network to have static IPs for each container and to pass this IP into the container as an environment - as I need it there for some RMI stuff configuration.

Now, I removed the custom network, and used the following code within my entrypoint to find out the dynamically assigned IP and write it into some variable:

setlocal enabledelayedexpansion
rem throw away everything except the IPv4 address line
for /f "usebackq tokens=*" %%a in (`ipconfig ^| findstr /i "ipv4"`) do (
  rem we have for example "IPv4 Address. . . . . . . . . . . : 192.168.42.78"
  rem split on : and get 2nd token
  for /f delims^=^:^ tokens^=2 %%b in ('echo %%a') do (
    rem we have " 192.168.42.78"
	set IP_ADDRESS=%%b
	set IP_ADDRESS=!IP_ADDRESS:~1!
    )
  )
)
(
endlocal
set IP_ADDRESS=%IP_ADDRESS%
)

Credits to this article on superuser

Answer 52 · 2019-02-13T03:40:08.000Z

While playing around with something completely different, I found something possibly related.

The host names are injected into the container by this automatically executed docker command (taken from the Windows Event Log with docker daemon debug set to true):
(192.168.XXX.XXX is the host IP)

debug: exec commandLine: cmd.exe /C "ECHO 192.168.XXX.XXX host.docker.internal >> %systemroot%\system32\drivers\etc\hosts & ECHO 192.168.XXX.XXX gateway.docker.internal >> %systemroot%\system32\drivers\etc\hosts" [...]

The POST call that triggered this command was this:

debug: form data: {"AttachStderr":true,"AttachStdin":false,"AttachStdout":true,"Cmd":["cmd.exe","/C","ECHO 192.168.XXX.XXX host.docker.internal \u003e\u003e %systemroot%\\system32\\drivers\\etc\\hosts \u0026 ECHO 192.168.XXX.XXX gateway.docker.internal \u003e\u003e %systemroot%\\system32\\drivers\\etc\\hosts"],"Detach":false,"DetachKeys":"","Env":null,"Privileged":false,"Tty":false,"User":"Administrator","WorkingDir":""}

The key part is "User":"Administrator".
On nanoserver:1803, the built-in Administrator account is enabled with no password and it works fine.
On servercore:1803, the built-in Administrator account is disabled (with a password) and it rejects the command.

CreateProcess: failure in a Windows system call: The user name or password is incorrect. (0x52e) [Event Detail: Provider: 00000000-0000-0000-0000-000000000000] [Event Detail: Provider: 00000000-0000-0000-0000-000000000000] [Event Detail: onecore\vm\compute\management\orchestration\vmhostedcontainer\processmanagement.cpp(174)\vmcomputeagent.exe!00007FF7B648C00A: (caller: 00007FF7B645ECEA) Exception(2) tid(36c) 8007052E The user name or password is incorrect. CallContext:[\Bridge_ProcessMessage\ComputeSystemManager_ExecuteProcess\VmHostedContainer_ExecuteProcess] Provider: 00000000-0000-0000-0000-000000000000]

Steps to reproduce
docker run --rm -it mcr.microsoft.com/windows/nanoserver:1803
(in container)
type %systemroot%\system32\drivers\etc\hosts
(Notice the hosts are listed correctly)

docker run --rm -it mcr.microsoft.com/windows/servercore:1803
(in container)
type %systemroot%\system32\drivers\etc\hosts
(Notice the hosts are missing)

Strangely, when I tried putting the "type ...hosts" directly after docker run, the hosts were missing from BOTH
docker run --rm -it mcr.microsoft.com/windows/[flavor]:1803 cmd /C type %systemroot%\system32\drivers\etc\hosts
(Notice the hosts are missing)

Unfortunately, this doesn't seem to actually fix the ping, merely the lookup.

Still, Docker should probably use "User":"ContainerAdministrator" instead of "User":"Administrator" when adding the hosts, at least on 1803. Unfortunately I can't test either flavor of 1809 to see if they behave the same way.

As a side note, if anyone wants to get the hosts working on servercore:

EDIT: In case it's not clear, this gives Administrator (aka root) a blank password, just like nanoserver. Be sure you understand the security implications.

# escape=`

FROM mcr.microsoft.com/windows/servercore:1803

SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

RUN Enable-LocalUser Administrator; `
    SecEdit.exe /export /cfg secpol.cfg; `
    (Get-Content secpol.cfg).Replace('PasswordComplexity = 1', 'PasswordComplexity = 0') | `
    Out-File secpol.cfg; `
    SecEdit.exe /configure /db C:\Windows\Security\Local.sdb /cfg secpol.cfg /areas SECURITYPOLICY; `
    Remove-Item secpol.cfg; `
    Set-LocalUser Administrator -Password (New-Object SecureString)

Answer 53 · 2019-02-18T08:32:30.000Z

RUN Enable-LocalUser Administrator; SecEdit.exe /export /cfg secpol.cfg;
(Get-Content secpol.cfg).Replace('PasswordComplexity = 1', 'PasswordComplexity = 0') | Out-File secpol.cfg;
SecEdit.exe /configure /db C:\Windows\Security\Local.sdb /cfg secpol.cfg /areas SECURITYPOLICY; Remove-Item secpol.cfg;
Set-LocalUser Administrator -Password (New-Object SecureString)

We have an internal tool that is run within a container used for our CI builds based on 1803 that seems to be affected by the absence of an administrator account.
Using the ps-script above we were able to get our container running again. But here is the peculiar thing:

The problem (CreateProcess: failure in a Windows system call: The user name or password is incorrect. (0x52e)) occurs after a period of about two weeks after installing docker to the machine the first time or going back to factory defaults. Either of the two fixes the problem but once a certain period has passed the issue reappears
When the issue reappears, using the powershell script above fixes the problem for a certain time period
Once this has been executed on this machine in one container and by using "--rm", it works for all other containers. So we do not have to execute the powershell script each time we want a working container but apparently only once within the above mentioned period. So there seems to be some kind of escaping the sandbox and usage of a cache on the docker host here. We could also observe that without the powershell script, the hosts file still contains two entries when the container is working and no entries when it is not working.

EDIT:

we suspect that the two weeks interval comes from a new dynamic ip adress of the docker host, this requires the hosts file to be altered as it contains the docker host ip adress
changing the docker host file from inside a container seems to require an administrator account but only then
updating the hosts file within the docker container seems to update the hosts file of the docker host and thus fixes the problem for a certain time period for all other containers as well

EDIT 2:

observed with Docker Desktop 2.0.0.3 / Engine 18.09.2

Answer 54 · 2019-02-19T11:05:14.000Z

Unfortunately, it was not just the dynamic ip address. We could not reproduce when our tool is causing the issue and when it is not.

Answer 55 · 2019-03-12T10:32:36.000Z

It's a windows daemon limitation and needs to be fix there but we will add a workaround waiting MS to fix it upstream.

@ebriney Is this still the case? Can we track the MS issue anywhere? Thanks

Answer 56 · 2019-03-19T17:02:00.000Z

Getting this issue

PS> docker run --rm -i microsoft/nanoserver:1709 ping host.docker.internal
Ping request could not find host host.docker.internal. Please check the name and try again.

Docker version

Client: Docker Engine - Community
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        6247962
 Built:             Sun Feb 10 04:12:31 2019
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.24)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 04:28:48 2019
  OS/Arch:          windows/amd64
  Experimental:     true

Answer 57 · 2019-04-04T02:07:48.000Z

I have the same issue with my container based on microsoft/aspnet:4.7.2-windowsservercore-1803. Trying to use host.docker.internal as the server for a db connection string to a sql database running locally on my host fails due to not being able to resolve host.docker.internal. I am also able to confirm that the host file inside the running container does not have records for host.docker.internal, nor for gateway.docker.internal. They are created and updated on my host machine's host file properly.

Please advise on a preferred workaround or a fix timetable.

PS C:\WINDOWS\system32> docker info
Containers: 13
 Running: 3
 Paused: 0
 Stopped: 10
Images: 15
Server Version: 18.09.2
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: ics l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 17134 (17134.1.amd64fre.rs4_release.180410-1804)
Operating System: Windows 10 Pro Version 1803 (OS Build 17134.677)
OSType: windows
Architecture: x86_64
CPUs: 8
Total Memory: 15.88GiB
Name: OH-DUB-LAP-48XS
ID: YEOQ:S5SV:YTJS:FJRW:CCE2:HZGV:SF2V:BXVR:AXZZ:P5AT:736C:KQZD
Docker Root Dir: C:\ProgramData\Docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 53
 System Time: 2019-04-03T22:05:04.3714794-04:00
 EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

Answer 58 · 2019-05-09T02:00:27.000Z

I can confirm I can reproduce on 18.09.2 with a windows server core image, more precisely: image 4.7.2-20190409-windowsservercore-ltsc2019.
Workaround script provided by @MagicShoebox fixes the issue.

Answer 59 · 2019-05-11T20:11:50.000Z

Is this still not fixed? It's 2019 guys

Answer 60 · 2019-05-13T21:08:27.000Z

Why can the docker-compose/swarm style DNS server not be used here? The A record can be managed by the container host instead of a host file. All docker containers get this DNS server entry first with the host's DNS servers appended. Just set the A record to the primary IPv4 address of the host.

Answer 61 · 2019-05-13T21:10:23.000Z

Still broke for me too:

PS > docker run --rm -i microsoft/nanoserver:1709 ping host.docker.internal
Unable to find image 'microsoft/nanoserver:1709' locally
1709: Pulling from microsoft/nanoserver
407ada6e90de: Already exists
9ef95ce817ec: Pull complete
Digest: sha256:97272be4e3abcd71727095a6ab1948afc4c8412ad0649abfe60912651769a67d
Status: Downloaded newer image for microsoft/nanoserver:1709
Ping request could not find host host.docker.internal. Please check the name and try again.

PS > docker info
Containers: 6
 Running: 0
 Paused: 0
 Stopped: 6
Images: 6
Server Version: 18.09.2
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: ics l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 16299 (16299.637.amd64fre.rs3_release_svc.180808-1748)
Operating System: Windows 10 Enterprise Version 1709 (OS Build 16299.1029)
OSType: windows
Architecture: x86_64
CPUs: 8
Total Memory: 63.89GiB
Name: xxxxxxx
ID: QQQQ:37AP:xxxx:MLOQ:Z4FE:xxxx:yyyy:RRYJ:xxxx:ZCCR:KKKK:xxxx
Docker Root Dir: C:\ProgramData\Docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 39
 System Time: 2019-05-13T16:54:40.6304645-04:00
 EventsListeners: 2
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

I'll note that it was working but I had an extra 20GB in windowsfilter that I couldn't get rid of so I did a "factory reset". After that a running container could not resolve DNS, I could tracert my host by ip, but not by name. I brought down another (unrelated) vm I had running in hyperv, reset docker desktop and found this command still failed:

PS C:\Users\grgran\docker\nanops> docker run --rm -i microsoft/nanoserver:1709 ping host.docker.internal
Ping request could not find host host.docker.internal. Please check the name and try again.

However, it did start resolving DNS and I could tracert www.google.com without issues. Don't know if this helps but it appears ping'ing host.docker.internal is not a reliable test.

Answer 62 · 2019-05-28T14:05:02.000Z

Pinging host.docker.internal is not a reliable test. A true DNS test would be to ping another docker-compose service.

The issue is present on 10.0.16299, and I confirm it works on 10.0.17763.

Answer 63 · 2019-07-05T04:17:28.000Z

Pinging host.docker.internal is not a reliable test. A true DNS test would be to ping another docker-compose service.
The issue is present on 10.0.16299, and I confirm it works on 10.0.17763.

agree: services of docker stack/swarm/compose, cannot ping to each other‘ internal DNS name from different swarm node.
but also disagree: not working on 10.0.17763.253, confirem as above ↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑

Answer 64 · 2019-08-07T13:25:56.000Z

Is there any update?
It's still not working.

Answer 65 · 2019-08-08T06:50:00.000Z

I don't believe it will ever be. Try to compile the windows binary yourself, copy it over the other one and it should work. https://github.com/rgl/docker-ce-windows-binaries-vagrant

Answer 66 · 2019-08-11T01:37:08.000Z

Any update on this? After it has been resolved for some time, the problem is back after the latest update to 2.1.0.1 (37199).

Answer 67 · 2019-08-11T11:57:59.000Z

Reinstallation of Docker for Windows solved the problem.

Answer 68 · 2019-09-05T06:58:58.000Z

I am still seeing the problem, I see people have commented about the hostfile, the hostfile has an entry on the docker machine but not on the container machine so will it work on the container?
I have tried reinstalling restarting, resetting etc.

without this you cant have part of the solution working locally and part of it on docker containers.

Answer 69 · 2019-09-17T18:16:41.000Z

The problem still persist in our environment (moby 39110) since April. Does anyone have a fix for us?

Answer 70 · 2019-09-27T10:19:29.000Z

Solution approach which seems to work is to write the gateway IP address (from within the container) to "drivers\etc\hosts" files, to make DNS resolving of "host.docker.internal" and "gateway.docker.internal" possible.

As the PS script from @kierzo has not worked for me using PowerShell Core

#1976 (comment)
due to:

Most of the modules that ship as part of Windows (for example, DnsClient, Hyper-V, NetTCPIP, Storage, etc.) and other Microsoft products including Azure and Office have not been explicitly ported to .NET Core yet.
https://docs.microsoft.com/en-us/powershell/scripting/whats-new/what-s-new-in-powershell-core-60?view=powershell-6

here is a minimum version working for PWSH Core v6 (for me):

$hostsFile = "${env:windir}\system32\drivers\etc\hosts"
$ip = (ipconfig | where-object {$_ –match "Default Gateway"} | foreach-object{$_.Split(":")[1]})
$addtohost = "`n#Added by Docker for Windows`n$ip    host.docker.internal`n$ip    gateway.docker.internal`n# End of section`n"
echo $addtohost | Out-File -Append -filepath $hostsFile

added it to our entrypoint.ps1 script and from there one used host.docker.internal to connect to services running on host (outside container).

Now back to work after spending an hour to dig through all posts and options around this issue :)

Waiting for a proper solution from MSFT to replace this hack.

Answer 71 · 2019-09-29T20:34:51.000Z

I am experiencing a random docker build failures (~40 minutes long builds on a build machine) and it seems that those somehow connected with this error (whole build fails because of the failed ECHO command ?)

[23:03:06.612][WindowsDaemon     ][Info   ] debug: start() completed [module=libcontainerd namespace=moby container=eb719a6ee48e274ae06cc35bff74e1d8e9d4c93dd
04ef0c2d3e32039e3823ba0]
[23:03:06.612][WindowsDaemon     ][Info   ] sending event [module=libcontainerd namespace=moby event-info={eb719a6ee48e274ae06cc35bff74e1d8e9d4c93dd04ef0c2d3
e32039e3823ba0 init 25660 0 0001-01-01 00:00:00 +0000 UTC false <nil>} container=eb719a6ee48e274ae06cc35bff74e1d8e9d4c93dd04ef0c2d3e32039e3823ba0 event=start
]
[23:03:06.612][WindowsDaemon     ][Info   ] debug: Calling HEAD /_ping
[23:03:06.612][WindowsDaemon     ][Info   ] debug: Calling GET /v1.40/containers/eb719a6ee48e274ae06cc35bff74e1d8e9d4c93dd04ef0c2d3e32039e3823ba0/json
[23:03:06.612][WindowsDaemon     ][Info   ] debug: Calling POST /v1.40/containers/eb719a6ee48e274ae06cc35bff74e1d8e9d4c93dd04ef0c2d3e32039e3823ba0/exec
[23:03:06.612][WindowsDaemon     ][Info   ] debug: form data: {"AttachStderr":true,"AttachStdin":false,"AttachStdout":true,"Cmd":["cmd.exe","/C","ECHO 10.128
.2.243    host.docker.internal \u003e\u003e %systemroot%\\system32\\drivers\\etc\\hosts \u0026 ECHO 10.128.2.243    gateway.docker.internal \u003e\u003e %sys
temroot%\\system32\\drivers\\etc\\hosts"],"Detach":false,"DetachKeys":"","Env":null,"Privileged":false,"Tty":false,"User":"Administrator","WorkingDir":""}
[23:03:06.612][WindowsDaemon     ][Info   ] debug: Calling POST /v1.40/exec/46dde09293222d87ac30f356167f1074138b34f064a8b8e9f6db44e85a4388b0/start
[23:03:06.612][WindowsDaemon     ][Info   ] debug: form data: {"Detach":false,"Tty":false}
[23:03:06.612][WindowsDaemon     ][Info   ] debug: starting exec command 46dde09293222d87ac30f356167f1074138b34f064a8b8e9f6db44e85a4388b0 in container eb719a
6ee48e274ae06cc35bff74e1d8e9d4c93dd04ef0c2d3e32039e3823ba0
[23:03:06.612][WindowsDaemon     ][Info   ] debug: exec commandLine: cmd.exe /C "ECHO 10.128.2.243    host.docker.internal >> %systemroot%\system32\drivers\e
tc\hosts & ECHO 10.128.2.243    gateway.docker.internal >> %systemroot%\system32\drivers\etc\hosts" [exec=46dde09293222d87ac30f356167f1074138b34f064a8b8e9f6d
b44e85a4388b0 module=libcontainerd namespace=moby container=eb719a6ee48e274ae06cc35bff74e1d8e9d4c93dd04ef0c2d3e32039e3823ba0]
[23:03:06.612][WindowsDaemon     ][Info   ] debug: hcsshim::ComputeSystem::CreateProcess - Begin Operation [cid=eb719a6ee48e274ae06cc35bff74e1d8e9d4c93dd04ef
0c2d3e32039e3823ba0]
[23:03:06.612][WindowsDaemon     ][Info   ] debug: HCS ComputeSystem Process Document [cid=eb719a6ee48e274ae06cc35bff74e1d8e9d4c93dd04ef0c2d3e32039e3823ba0 j
son={"CommandLine":"cmd.exe /C \"ECHO 10.128.2.243    host.docker.internal \u003e\u003e %systemroot%\\system32\\drivers\\etc\\hosts \u0026 ECHO 10.128.2.243
   gateway.docker.internal \u003e\u003e %systemroot%\\system32\\drivers\\etc\\hosts\"","User":"Administrator","WorkingDirectory":"C:\\build","Environment":{"
COMPLUS_NGenProtectedProcess_FeatureEnabled":"0","GIT_ASKPASS":"C:\\git-credential-helper.bat","HOST_ADDRESS":"10.128.2.243","HOST_TOKEN":"683cb9d70552cbf762
20861aa96ba28f","NUGET_VERSION":"4.4.1","ROSLYN_COMPILER_LOCATION":"C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\MSBuild\\Current\\Bin
\\Roslyn"},"CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]}]
[23:03:06.612][WindowsDaemon     ][Info   ] debug: attach: stderr: begin
[23:03:06.612][WindowsDaemon     ][Info   ] debug: attach: stdout: begin
[23:03:06.612][WindowsDaemon     ][Info   ] debug: HCS Result [json={"Error":-2147023570,"ErrorMessage":"The user name or password is incorrect."}]
[23:03:06.612][WindowsDaemon     ][Error  ] hcsshim::ComputeSystem::CreateProcess - End Operation - Error [cid=eb719a6ee48e274ae06cc35bff74e1d8e9d4c93dd04ef0
c2d3e32039e3823ba0 error=CreateProcess eb719a6ee48e274ae06cc35bff74e1d8e9d4c93dd04ef0c2d3e32039e3823ba0: The user name or password is incorrect.

and after that:

[23:03:06.613][WindowsDaemon     ][Error  ] Error running exec 46dde09293222d87ac30f356167f1074138b34f064a8b8e9f6db44e85a4388b0 in container: container eb719
a6ee48e274ae06cc35bff74e1d8e9d4c93dd04ef0c2d3e32039e3823ba0 encountered an error during CreateProcess: failure in a Windows system call: The user name or pas
sword is incorrect. (0x52e) extra info: {"CommandLine":"cmd.exe /C \"ECHO 10.128.2.243    host.docker.internal \u003e\u003e %systemroot%\\system32\\drivers\\
etc\\hosts \u0026 ECHO 10.128.2.243    gateway.docker.internal \u003e\u003e %systemroot%\\system32\\drivers\\etc\\hosts\"","User":"Administrator","WorkingDir
ectory":"C:\\build","Environment":{"COMPLUS_NGenProtectedProcess_FeatureEnabled":"0","GIT_ASKPASS":"C:\\git-credential-helper.bat","HOST_ADDRESS":"10.128.2.2
43","HOST_TOKEN":"683cb9d70552cbf76220861aa96ba28f","NUGET_VERSION":"4.4.1","ROSLYN_COMPILER_LOCATION":"C:\\Program Files (x86)\\Microsoft Visual Studio\\201
9\\BuildTools\\MSBuild\\Current\\Bin\\Roslyn"},"CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]}
[23:03:06.613][WindowsDaemon     ][Info   ] debug: attach: stdout: end
[23:03:06.613][WindowsDaemon     ][Info   ] debug: attach: stderr: end
[23:03:06.613][WindowsDaemon     ][Info   ] debug: attach done
[23:03:06.613][WindowsDaemon     ][Info   ] debug: Calling GET /v1.40/exec/46dde09293222d87ac30f356167f1074138b34f064a8b8e9f6db44e85a4388b0/json
[23:03:06.613][WindowsDaemon     ][Info   ] debug: clean 1 unused exec commands
[23:03:06.613][WindowsDaemon     ][Info   ] debug: Client context cancelled, stop sending events
[23:03:06.613][WindowsDaemon     ][Info   ] debug: Calling GET /v1.40/events
[23:03:06.613][WindowsDaemon     ][Info   ] debug: clean 3 unused exec commands
[23:03:06.613][WindowsDaemon     ][Info   ] debug: Client context cancelled, stop sending events
[23:03:06.613][WindowsDaemon     ][Info   ] debug: Calling GET /v1.40/events
[23:03:06.613][WindowsDaemon     ][Info   ] debug: Client context cancelled, stop sending events
[23:03:06.613][WindowsDaemon     ][Info   ] debug: Calling GET /v1.40/events
[23:03:06.613][WindowsDaemon     ][Info   ] debug: Build cancelled, killing and removing container: eb719a6ee48e274ae06cc35bff74e1d8e9d4c93dd04ef0c2d3e32039e
3823ba0
[23:03:06.614][WindowsDaemon     ][Info   ] debug: Sending kill signal 9 to container eb719a6ee48e274ae06cc35bff74e1d8e9d4c93dd04ef0c2d3e32039e3823ba0

and the result is:

[23:03:07.641][WindowsDaemon     ][Info   ] debug: hcsshim::DestroyLayer - succeeded [path=D:\docker_windows\windowsfilter\0c92737e7b38f61d3bb4aed66d69280aa7
89283a35da07f5ce7802784ca8f6a6-removing]
[23:03:07.641][WindowsDaemon     ][Info   ] debug: Build cancelled (build cancelled): a non-zero code from ContainerWait: 3221225473

I've tried to apply @MagicShoebox script, but with no luck.

> docker info
Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., v0.3.0-5-g5b97415-tp-docker)

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 115
 Server Version: 19.03.2
 Storage Driver: windowsfilter (windows) lcow (linux)
  Windows:
  LCOW:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics l2bridge l2tunnel nat null overlay transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: process
 Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
 Operating System: Windows 10 Enterprise Version 1809 (OS Build 17763.737)
 OSType: windows
 Architecture: x86_64
 CPUs: 12
 Total Memory: 63.95GiB
 Name: BY1-CABuilds-01
 ID: RS5M:6DEO:RNHL:MYF4:CBRW:T6XD:E6CT:HMVF:HRXN:FSKU:MSJN:5GRM
 Docker Root Dir: D:\docker_windows
 Debug Mode: true
  File Descriptors: -1
  Goroutines: 33
  System Time: 2019-09-29T23:25:41.1348512+03:00
  EventsListeners: 1
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Answer 72 · 2019-10-03T16:06:54.000Z

I am updating the hosts file when the container starts but there is an issue with that because the ip resolved in a container where hosts.docker.internal works is different from the gateway ip address used in the workaround.

Answer 73 · 2019-10-17T13:31:24.000Z

I think I found the cause of the issue, when looking at the Docker logs for a different issue I found this...

[Event Detail: onecore\vm\compute\management\orchestration\vmhostedcontainer\processmanagement.cpp(173)\vmcomputeagent.exe!00007FF6000D9ADB: (caller: 00007FF60008DF1A) Exception(2) tid(4c4) 8007052E The user name or password is incorrect.
    CallContext:[\Bridge_ProcessMessage\VmHostedContainer_ExecuteProcess] 
 Provider: 00000000-0000-0000-0000-000000000000]
(extra info: {"CommandLine":"cmd.exe /C \"ECHO 192.168.1.204    host.docker.internal \u003e\u003e %systemroot%\\system32\\drivers\\etc\\hosts \u0026 ECHO 192.168.1.204    gateway.docker.internal \u003e\u003e %systemroot%\\system32\\drivers\\etc\\hosts\"","User":"Administrator","WorkingDirectory":"/","Environment":{"COMPLUS_NGenProtectedProcess_FeatureEnabled":"0","ROSLYN_COMPILER_LOCATION":"c:\\\\RoslynCompilers\\\\tools"},"CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]})]

The important thing to note is the error The user name or password is incorrect and the command seems to be trying to add host.docker.internal into the hosts file.

Answer 74 · 2019-11-01T15:14:55.000Z

I have been successfully building windows server images without issue on Windows 10 Enterprise Insiders edition for the past month or two, last night I upgraded to the latest Win10 insiders edition and this DNS resolving issue has begun to manifest itself. I've submitted an insiders bug report.

Note: copying in a fake hosts file fixed the issue, however that isn't a viable solution as most of you have already found.

Answer 75 · 2020-01-30T01:00:16.000Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

Answer 76 · 2020-01-30T08:48:06.000Z

what is the status of this issue?

Answer 77 · 2020-01-30T09:35:51.000Z

They probably don't want to fix it

Answer 78 · 2020-01-31T19:47:13.000Z

/remove-lifecycle stale

Answer 79 · 2020-02-06T16:55:35.000Z

For what it's worth, it's not working in my environment either:

Windows 10 Pro
Docker version 2.1.0.5 (40693)
Docker Engine 19.03.5

Executing
docker exec -it CONTAINERID ping docker.host.internal
returns
Ping request could not find host docker.host.internal. Please check the name and try again.

Same happens with "docker.gateway.internal".

Executing
docker exec -it CONTAINERID ping www.google.com
works fine.

I agree with previous comments stating that adding an entry to the hosts file may not be appropriate; in our case we may lose connectivity and reconnect later with a different IP. Though I'm not much of a network expert.

Answer 80 · 2020-02-12T07:47:09.000Z

Executing
docker exec -it CONTAINERID ping docker.host.internal
returns
Ping request could not find host docker.host.internal. Please check the name and try again.

Same happens with "docker.gateway.internal".

I guess it's host.docker.internal, not docker.host.internal...

Answer 81 · 2020-02-12T08:33:09.000Z

I guess it's host.docker.internal, not docker.host.internal...

@llinke I guess you're so very right. Thank you for taking the time to answer (and apologies for me not being able to read words in the right order).

Answer 82 · 2020-02-12T21:20:08.000Z

I have some Windows containers that were resolving host.docker.internal and gateway.docker.internal, along with some other Windows containers that weren't resolving those names, all in the same docker-compose stack.

After the last update, none of them are working anymore for some reason, so the workaround I did (based on some comments in this same issue) is this:

Dockerfile:

ENTRYPOINT ["powershell", "/docker-entrypoint.ps1;"]

Entrypoint script:

# DNS Entries
Write-Host("Configuring DNS entries...")
C:\fix-hosts.ps1

fix-hosts.ps1:

$hostsFile = "C:\Windows\System32\drivers\etc\hosts"

try {
    $DnsEntries = @("host.docker.internal", "gateway.docker.internal")
    # Tries resolving names for Docker
    foreach ($Entry in $DnsEntries) {
        # If any of the names are not resolved, throws an exception
        Resolve-DnsName -Name $Entry -ErrorAction Stop
    }

    # If it passes, means that DNS is already configured
    Write-Host("DNS settings are already configured.")
} catch {
    # Gets the gateway IP address, that is the Host's IP address in the Docker network
    $ip = (ipconfig | where-object {$_ -match "Default Gateway"} | foreach-object{$_.Split(":")[1]}).Trim()
    # Read the current content from Hosts file
    $src = [System.IO.File]::ReadAllLines($hostsFile)
    # Add the a new line after the content
    $lines = $src += ""
    
    # Check the hosts file and write it in if its not there...
    if((cat $hostsFile | Select-String -Pattern "host.docker.internal") -And (cat $hostsFile | Select-String -Pattern "gateway.docker.internal")) {
        For ($i=0; $i -le $lines.length; $i++) {
            if ($lines[$i].Contains("host.docker.internal"))
            {
                $lines[$i] = ("{0} host.docker.internal" -f $ip)
                $lines[$i+1] = ("{0} gateway.docker.internal" -f $ip)
                break
            }
        }
    } else {
        $lines = $lines += "# Added by Docker for Windows"
        $lines = $lines += ("{0} host.docker.internal" -f $ip)
        $lines = $lines += ("{0} gateway.docker.internal" -f $ip)
        $lines = $lines += "# End of section"
    }
    # Writes the new content to the Hosts file
    [System.IO.File]::WriteAllLines($hostsFile, [string[]]$lines)
    
    Write-Host("New DNS settings written successfully.")
}

When the container starts, before executing the main command it starts the fix-hosts.ps1 script.

What the script does is:

Checks if the names can be resolved.
- If they can, assumes it's configured properly.
If not, looks up in the hosts file if there are entries there already.
- If there is, replace them with the gateway IP address, which is the host's IP address.
If not, just adds the new entries for both gateway.docker.internal and host.docker.internal in the hosts file.
Write the new content to the hosts file.
The end.

That's it! It's the neatest approach I could get so far without having to hack it too much, and it works with pretty much any Windows image.

Answer 83 · 2020-03-05T13:04:13.000Z

Are you using AzureAD by any chance? i had the same problem and it was a permission problem, i fixed it with this script :
$path = "C:\ProgramData\docker" $permission = "AzureAD\you@yourcompany.com","FullControl","Allow" (Get-Acl $path).SetAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule $permission)) | Set-Acl $path

Answer 84 · 2020-03-10T14:28:00.000Z

I am seeing the same results as @brianharwell. I can see "The user name or password is incorrect." in the docker daemon logs. This is happening when the attempt is made to add 'host.docker.internal' to the container hosts file. Note the User is Administrator.

This only fails with servercore (FROM mcr.microsoft.com/windows/servercore).

If I use exactly the same docker file but change to nanoserver (FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-nanoserver-1809) it works perfectly. I can see in the docker daemon log the hosts file being successfully updated with host.docker.internal.

The User is showing as Administrator in both log files.

I can only conclude that docker is trying to pass the same password to the servercore and nanoserver images but the passwords are in fact not the same on both images?

Below is the extract from the failed servercore daemon log startup,

[08:36:16.473][WindowsDaemon ][Info ] debug: HCS ComputeSystem Process Document [cid=17d0c4096b6e94ba507f867283e17d54aa4270643cc29945904ba1f9c6bde0d1 json={"CommandLine":"cmd.exe /C \"ECHO 10.3.112.207 host.docker.internal \u003e\u003e %systemroot%\\system32\\drivers\\etc\\hosts \u0026 ECHO 10.3.112.207 gateway.docker.internal \u003e\u003e %systemroot%\\system32\\drivers\\etc\\hosts\"","User":"Administrator","WorkingDirectory":"C:\\app","Environment":{"ASPNETCORE_URLS":"http://+:80","DOTNET_RUNNING_IN_CONTAINER":"true","DOTNET_SDK_VERSION":"3.1.100","DOTNET_USE_POLLING_FILE_WATCHER":"true","NUGET_XMLDOC_MODE":"skip"},"CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]}] [08:36:16.527][WindowsDaemon ][Info ] debug: HCS Result [json={"Error":-2147023570,"ErrorMessage":"The user name or password is incorrect.","ErrorEvents":[{"Message":"","Provider":"00000000-0000-0000-0000-000000000000","EventId":0},{"Message":"","Provider":"00000000-0000-0000-0000-000000000000","EventId":0},{"Message":"onecore\\vm\\compute\\management\\orchestration\\vmhostedcontainer\\processmanagement.cpp(173)\\vmcomputeagent.exe!00007FF64CB29D2B: (caller: 00007FF64CADE13A) Exception(2) tid(4dc) 8007052E The user name or password is incorrect.\r\n CallContext:[\\Bridge_ProcessMessage\\VmHostedContainer_ExecuteProcess] \n","Provider":"00000000-0000-0000-0000-000000000000","EventId":0}]}]

Answer 85 · 2020-05-12T20:31:03.000Z

I've having the same problem with FROM mcr.microsoft.com/windows/servercore:1803. It works with nanoserver:1803, but I need powershell for my images and nanoserver:1803 doesn't have powershell. If I try to pull mcr.microsoft.com/powershell, the image isn't compatible with my version of Windows 10 Pro. So, I'm kind of stuck. If @karlholbrook is correct, this seems like it would be an easy fix on MS's side to correct the password issue in servercore:1803

Answer 86 · 2020-05-15T02:23:26.000Z

Have some ASP apps I would like to containerize which rely on SQL Server. This is a blocker for local development, do we know where the issue lies?

Answer 87 · 2020-07-02T09:30:33.000Z

I've had issues with host.docker.internal created accidentally by me.
The issue was that I needed to fix another issue related to nodejs stucking while installing dependencies. The issue was resolved by adding a second DNS to already available 8.8.8.8. The dependencies installation started working but apparently that caused this issue with host.docker.internal.

So I traced back what I've done recently and remove the additional DNS and left only 8.8.8.8. Restarted Docker and now works fine.

Although, there will be an issue again with nodejs, so this somehow needs to be resolved.

My Docker info is:
Version: 2.3.0.3
Engine: 19.03.8

Answer 88 · 2020-07-06T00:08:34.000Z

Still seeing this behaviour. Windows 10, ServerCore.

Answer 89 · 2020-10-04T01:00:23.000Z

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

Answer 90 · 2020-10-05T09:34:24.000Z

/remove-lifecycle stale

Answer 91 · 2021-01-03T01:00:27.000Z

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

Answer 92 · 2021-01-04T08:13:37.000Z

/remove-lifecycle stale

Answer 93 · 2021-02-05T02:39:55.000Z

Still getting this issue, my containers (same images) randomly fail on startup. sometimes they start, sometimes they hang in "created" state and application log filled with repeated errors like below. I am running windows containers on windows 10 surface book laptop, calling docker run from PowerShell ISE elevated (running as admin) and still hit this issue at least 50% of the time, without changing anything in my images:

exec's CreateProcess() failed [module=libcontainerd namespace=moby container=a06ed0f5c4f8b94c5462d4d64c5b370e4db7cc95422dda325f1fc44abf186a6c exec=a6ccdf64ab7e3b48360dc82630cbb028263e1232ace044f1285d2b71e3a489da error=container a06ed0f5c4f8b94c5462d4d64c5b370e4db7cc95422dda325f1fc44abf186a6c encountered an error during hcsshim::System::CreateProcess: failure in a Windows system call: The user name or password is incorrect. (0x52e)
[Event Detail:  Provider: 00000000-0000-0000-0000-000000000000]
[Event Detail:  Provider: 00000000-0000-0000-0000-000000000000]
[Event Detail: onecore\vm\compute\management\orchestration\vmhostedcontainer\processmanagement.cpp(173)\vmcomputeagent.exe!00007FF7E2119D2B: (caller: 00007FF7E20CE13A) Exception(2) tid(4d4) 8007052E The user name or password is incorrect.
    CallContext:[\Bridge_ProcessMessage\VmHostedContainer_ExecuteProcess] 
 Provider: 00000000-0000-0000-0000-000000000000] extra info: {"CommandLine":"cmd.exe /C \"ECHO 192.168.0.117    host.docker.internal \u003e\u003e %systemroot%\\system32\\drivers\\etc\\hosts \u0026 ECHO 192.168.0.117    gateway.docker.internal \u003e\u003e %systemroot%\\system32\\drivers\\etc\\hosts\"","User":"Administrator","WorkingDirectory":"/","CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]}]

Answer 94 · 2021-02-05T13:03:23.000Z

Is anyone looking into the root cause of this?

Answer 95 · 2021-02-26T15:54:10.000Z

/remove-lifecycle stale

Answer 96 · 2021-05-11T09:29:22.000Z

Docker 3.3.3 Same issue

Answer 97 · 2021-05-15T08:51:25.000Z

Updated to Docker Desktop 3.3.3 a few days ago, and this issue seems to have crop up again (hostname gateway.docker.internal was resolving fine before then). Perhaps something broke in this latest update?

Answer 98 · 2021-05-28T01:59:20.000Z

I am on Docker 3.3.3 and I also am seeing this issue - anyone know a specific version of Docker where it actually did work?

Answer 99 · 2021-05-28T08:38:50.000Z

I am on "Docker version 20.10.6, build 370c289" and have the same problem. I've used @brunnotelma 's workaround successfully, thanks for that!

Answer 100 · 2021-05-28T16:48:08.000Z

If I use https://docs.docker.com/docker-for-windows/release-notes/#docker-desktop-community-2501 with docker.for.win.localhost or host.docker.internal I see the same issues - @brunnotelma's workaround did not fix this issue with Docker 3.x either. I am tracking this issue with reproducible images here: envoyproxy/envoy#16710 in case it is of interest to the Docker team