Cannot connect to containers when using docker-in-docker
jameslamine opened this issue · 10 comments
jameslamine commented
I'm using docker-in-docker inside of kubernetes in jenkins. My containers appear to start up fine, but the host/port never becomes accessible and so testcontainers cannot connect to them.
Everything works fine when I run the tests locally using docker desktop, it's only on when running in jenkins using docker-in-docker that they fail.
The container ip address is 172.17.0.2 but it's trying to connect to 172.17.0.1 which is the gateway. Is that normal?
I've tested this on both testcontainers 1.16.0 and 1.16.1 and they both fail. This could be related to #4594, except my problem happens on 1.16.0 as well. This also might be related to #4591 although I am not using TESTCONTAINERS_HOST_OVERRIDE
Here is my test case where I start up an elasticsearch container, log the ipAddress and hostAddress, and then run docker inspect and log the results:
class ElasticsearchContainerTest {
companion object {
private val logger = KontextLogger.logger {}
private const val ELASTICSEARCH_VERSION = "6.7.0"
private val container = ElasticsearchContainer(
DockerImageName.parse("{REDACTED CONTAINER MIRROR URL}/elasticsearch")
.withTag(ELASTICSEARCH_VERSION)
.asCompatibleSubstituteFor("docker.elastic.co/elasticsearch/elasticsearch")
).withLogConsumer(Slf4jLogConsumer(KontextLogger.logger {}))
@BeforeAll
@JvmStatic
fun beforeAll() {
container.start()
}
@AfterAll
@JvmStatic
fun afterAll() {
logger.error("Container IP address: ${container.containerIpAddress}")
logger.error("Container url: ${container.httpHostAddress}")
val dockerInspect = container.dockerClient.inspectContainerCmd(container.containerId).exec()
val mapper = ObjectMapper().disable(SerializationFeature.FAIL_ON_EMPTY_BEANS)
val jsonOutput = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(dockerInspect)
logger.error("docker inspect elasticsearch: $jsonOutput")
container.stop()
}
}
@Test
fun `check that container is running`() {
Assert.assertTrue("Container is not running", container.isRunning)
}
}
The test fails due to Timed out waiting for URL to be accessible (http://172.17.0.1:49154/ should return HTTP 200:
:integrationTest > 0 tests completed> :integrationTest > Executing test com.example.test.Elasticsea
ElasticsearchContainerTest > initializationError FAILED
org.testcontainers.containers.ContainerLaunchException: Container startup failed
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:336)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)
at com.example.test.ElasticsearchContainerTest$Companion.beforeAll(ElasticsearchContainerTest.kt:28)
at com.example.test.ElasticsearchContainerTest.beforeAll(ElasticsearchContainerTest.kt)
Caused by:
org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:329)
... 3 more
Caused by:
org.testcontainers.containers.ContainerLaunchException: Could not create/start container
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:525)
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:331)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
... 4 more
Caused by:
org.testcontainers.containers.ContainerLaunchException: Timed out waiting for URL to be accessible (http://172.17.0.1:49154/ should return HTTP 200)
at org.testcontainers.containers.wait.strategy.HttpWaitStrategy.waitUntilReady(HttpWaitStrategy.java:264)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:51)
at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:929)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:468)
... 6 more
Here is the output of docker inspect:
{
"Args" : [ "eswrapper" ],
"Config" : {
"healthcheck" : null,
"AttachStderr" : false,
"AttachStdin" : false,
"AttachStdout" : false,
"Cmd" : [ "eswrapper" ],
"Domainname" : "",
"Entrypoint" : [ "/usr/local/bin/docker-entrypoint.sh" ],
"Env" : [ "discovery.type=single-node", "PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "ELASTIC_CONTAINER=true", "JAVA_HOME=/opt/jdk-12" ],
"ExposedPorts" : {
"9200/tcp" : { },
"9300/tcp" : { }
},
"Hostname" : "8d8c72fac5a9",
"Image" : "{REDACTED CONTAINER MIRROR URL}/open-source-mirror/dockerhub/elasticsearch:6.7.0",
"Labels" : {
"license" : "Elastic License",
"org.label-schema.build-date" : "20190305",
"org.label-schema.license" : "GPLv2",
"org.label-schema.name" : "elasticsearch",
"org.label-schema.schema-version" : "1.0",
"org.label-schema.url" : "https://www.elastic.co/products/elasticsearch",
"org.label-schema.vcs-url" : "https://github.com/elastic/elasticsearch",
"org.label-schema.vendor" : "Elastic",
"org.label-schema.version" : "6.7.0",
"org.testcontainers" : "true",
"org.testcontainers.sessionId" : "671d3a01-4780-4f87-af21-ac568bafdc6f"
},
"MacAddress" : null,
"NetworkDisabled" : null,
"OnBuild" : null,
"OpenStdin" : false,
"PortSpecs" : null,
"StdinOnce" : false,
"Tty" : false,
"User" : "",
"Volumes" : null,
"WorkingDir" : "/usr/share/elasticsearch",
"Healthcheck" : null
},
"Created" : "2021-10-20T20:26:52.934006535Z",
"Driver" : "overlay2",
"ExecDriver" : null,
"HostConfig" : {
"Binds" : [ ],
"BlkioWeight" : 0,
"BlkioWeightDevice" : null,
"BlkioDeviceReadBps" : null,
"BlkioDeviceWriteBps" : null,
"BlkioDeviceReadIOps" : null,
"BlkioDeviceWriteIOps" : null,
"MemorySwappiness" : null,
"NanoCpus" : 0,
"CapAdd" : null,
"CapDrop" : null,
"ContainerIDFile" : "",
"CpuPeriod" : 0,
"CpuRealtimePeriod" : 0,
"CpuRealtimeRuntime" : 0,
"CpuShares" : 0,
"CpuQuota" : 0,
"CpusetCpus" : "",
"CpusetMems" : "",
"Devices" : null,
"DeviceCgroupRules" : null,
"DeviceRequests" : null,
"DiskQuota" : null,
"Dns" : null,
"DnsOptions" : null,
"DnsSearch" : null,
"ExtraHosts" : [ ],
"GroupAdd" : null,
"IpcMode" : "shareable",
"Cgroup" : "",
"Links" : null,
"LogConfig" : {
"Type" : "json-file",
"Config" : { }
},
"LxcConf" : null,
"Memory" : 0,
"MemorySwap" : 0,
"MemoryReservation" : 0,
"KernelMemory" : 0,
"NetworkMode" : "default",
"OomKillDisable" : false,
"Init" : null,
"AutoRemove" : false,
"OomScoreAdj" : 0,
"PortBindings" : {
"9200/tcp" : [ {
"HostIp" : "",
"HostPort" : ""
} ],
"9300/tcp" : [ {
"HostIp" : "",
"HostPort" : ""
} ]
},
"Privileged" : false,
"PublishAllPorts" : false,
"ReadonlyRootfs" : false,
"RestartPolicy" : {
"MaximumRetryCount" : 0,
"Name" : ""
},
"Ulimits" : null,
"CpuCount" : 0,
"CpuPercent" : 0,
"IOMaximumIOps" : 0,
"IOMaximumBandwidth" : 0,
"VolumesFrom" : [ ],
"Mounts" : null,
"PidMode" : "",
"Isolation" : null,
"SecurityOpt" : null,
"StorageOpt" : null,
"CgroupParent" : "",
"VolumeDriver" : "",
"ShmSize" : 67108864,
"PidsLimit" : null,
"Runtime" : "runc",
"Tmpfs" : null,
"UTSMode" : "",
"UsernsMode" : "",
"Sysctls" : null,
"ConsoleSize" : [ 0, 0 ]
},
"HostnamePath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hostname",
"HostsPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hosts",
"LogPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457-json.log",
"Id" : "8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457",
"SizeRootFs" : null,
"Image" : "sha256:02982be5777dea7bbf9573e1993c936b8add775d303bafd638924cf9635e8c38",
"MountLabel" : "",
"Name" : "/focused_roentgen",
"RestartCount" : 0,
"NetworkSettings" : {
"Bridge" : "",
"SandboxID" : "786b67295fe24a75cd4edcf66b69383e72e2972f80bf6b6a801f9c020d14bf27",
"HairpinMode" : false,
"LinkLocalIPv6Address" : "",
"LinkLocalIPv6PrefixLen" : 0,
"Ports" : {
"9200/tcp" : [ {
"HostIp" : "0.0.0.0",
"HostPort" : "49154"
}, {
"HostIp" : "::",
"HostPort" : "49154"
} ],
"9300/tcp" : [ {
"HostIp" : "0.0.0.0",
"HostPort" : "49153"
}, {
"HostIp" : "::",
"HostPort" : "49153"
} ]
},
"SandboxKey" : "/run/user/1000/docker/netns/786b67295fe2",
"SecondaryIPAddresses" : null,
"SecondaryIPv6Addresses" : null,
"EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
"Gateway" : "172.17.0.1",
"PortMapping" : null,
"GlobalIPv6Address" : "",
"GlobalIPv6PrefixLen" : 0,
"IPAddress" : "172.17.0.2",
"IPPrefixLen" : 16,
"IPv6Gateway" : "",
"MacAddress" : "02:42:ac:11:00:02",
"Networks" : {
"bridge" : {
"IPAMConfig" : null,
"Links" : null,
"Aliases" : null,
"NetworkID" : "aaeb65fa1c11b6e4e4350d4b68b34d38796f2170d26073bdb0282fe14f4dc6a6",
"EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
"Gateway" : "172.17.0.1",
"IPAddress" : "172.17.0.2",
"IPPrefixLen" : 16,
"IPv6Gateway" : "",
"GlobalIPv6Address" : "",
"GlobalIPv6PrefixLen" : 0,
"MacAddress" : "02:42:ac:11:00:02"
}
}
},
"Path" : "/usr/local/bin/docker-entrypoint.sh",
"ProcessLabel" : "",
"ResolvConfPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/resolv.conf",
"ExecIDs" : null,
"State" : {
"oomkilled" : false,
"pidLong" : 873,
"exitCodeLong" : 0,
"Status" : "running",
"Running" : true,
"Paused" : false,
"Restarting" : false,
"OOMKilled" : false,
"Dead" : false,
"Pid" : 873,
"ExitCode" : 0,
"Error" : "",
"StartedAt" : "2021-10-20T20:26:53.468541843Z",
"FinishedAt" : "0001-01-01T00:00:00Z",
"Health" : null
},
"Volumes" : null,
"VolumesRW" : null,
"Node" : null,
"Mounts" : [ ],
"GraphDriver" : {
"Name" : "overlay2",
"Data" : {
"RootDir" : null,
"DeviceId" : null,
"DeviceName" : null,
"DeviceSize" : null,
"dir" : null
}
},
"Platform" : "linux",
"Path" : "/usr/local/bin/docker-entrypoint.sh",
"Args" : [ "eswrapper" ],
"ProcessLabel" : "",
"Mounts" : [ ],
"Platform" : "linux",
"Config" : {
"Hostname" : "8d8c72fac5a9",
"Domainname" : "",
"User" : "",
"AttachStdin" : false,
"AttachStdout" : false,
"AttachStderr" : false,
"ExposedPorts" : {
"9200/tcp" : { },
"9300/tcp" : { }
},
"Tty" : false,
"OpenStdin" : false,
"StdinOnce" : false,
"Env" : [ "discovery.type=single-node", "PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "ELASTIC_CONTAINER=true", "JAVA_HOME=/opt/jdk-12" ],
"Cmd" : [ "eswrapper" ],
"Image" : "{REDACTED CONAINER MIRROR URL}/open-source-mirror/dockerhub/elasticsearch:6.7.0",
"Volumes" : null,
"WorkingDir" : "/usr/share/elasticsearch",
"Entrypoint" : [ "/usr/local/bin/docker-entrypoint.sh" ],
"OnBuild" : null,
"Labels" : {
"license" : "Elastic License",
"org.label-schema.build-date" : "20190305",
"org.label-schema.license" : "GPLv2",
"org.label-schema.name" : "elasticsearch",
"org.label-schema.schema-version" : "1.0",
"org.label-schema.url" : "https://www.elastic.co/products/elasticsearch",
"org.label-schema.vcs-url" : "https://github.com/elastic/elasticsearch",
"org.label-schema.vendor" : "Elastic",
"org.label-schema.version" : "6.7.0",
"org.testcontainers" : "true",
"org.testcontainers.sessionId" : "671d3a01-4780-4f87-af21-ac568bafdc6f"
}
},
"Driver" : "overlay2",
"AppArmorProfile" : "",
"HostsPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hosts",
"HostnamePath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hostname",
"GraphDriver" : {
"Data" : {
"LowerDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5-init/diff:/home/rootless/.local/share/docker/overlay2/1e71332ad2a429a4d06d157b6c4e2c4401da2b44b640d412557feb3c39452010/diff:/home/rootless/.local/share/docker/overlay2/1302968f5de6c81580eadb218555c0bc49bd3af7c766e77cbdb0ea9f02371d2a/diff:/home/rootless/.local/share/docker/overlay2/3b2b6c22a55172df81ac6af6f439aa3ec1cfe0576269d9fab8c25772e49a65ca/diff:/home/rootless/.local/share/docker/overlay2/42a8e273145d92d47a2c6827fa4d276800e98189da117ca69b25808ce1013edc/diff:/home/rootless/.local/share/docker/overlay2/be84a76283d29eac19bb5e262a4b26578218877d2302cea775fc57c29e195fb0/diff:/home/rootless/.local/share/docker/overlay2/05a94b779ac49607562a956de5a274e5cd000747ef966e612f156ba3f5018cb3/diff:/home/rootless/.local/share/docker/overlay2/57967e2c3b2027a4b67e5d97544d36b11573228ae195fb1cb0799b0ca954c669/diff",
"MergedDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5/merged",
"UpperDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5/diff",
"WorkDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5/work"
},
"Name" : "overlay2"
},
"Image" : "sha256:02982be5777dea7bbf9573e1993c936b8add775d303bafd638924cf9635e8c38",
"Created" : "2021-10-20T20:26:52.934006535Z",
"Name" : "/focused_roentgen",
"RestartCount" : 0,
"NetworkSettings" : {
"Bridge" : "",
"SandboxID" : "786b67295fe24a75cd4edcf66b69383e72e2972f80bf6b6a801f9c020d14bf27",
"HairpinMode" : false,
"LinkLocalIPv6Address" : "",
"LinkLocalIPv6PrefixLen" : 0,
"Ports" : {
"9200/tcp" : [ {
"HostIp" : "0.0.0.0",
"HostPort" : "49154"
}, {
"HostIp" : "::",
"HostPort" : "49154"
} ],
"9300/tcp" : [ {
"HostIp" : "0.0.0.0",
"HostPort" : "49153"
}, {
"HostIp" : "::",
"HostPort" : "49153"
} ]
},
"SandboxKey" : "/run/user/1000/docker/netns/786b67295fe2",
"SecondaryIPAddresses" : null,
"SecondaryIPv6Addresses" : null,
"EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
"Gateway" : "172.17.0.1",
"GlobalIPv6Address" : "",
"GlobalIPv6PrefixLen" : 0,
"IPAddress" : "172.17.0.2",
"IPPrefixLen" : 16,
"IPv6Gateway" : "",
"MacAddress" : "02:42:ac:11:00:02",
"Networks" : {
"bridge" : {
"IPAMConfig" : null,
"Links" : null,
"Aliases" : null,
"NetworkID" : "aaeb65fa1c11b6e4e4350d4b68b34d38796f2170d26073bdb0282fe14f4dc6a6",
"EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
"Gateway" : "172.17.0.1",
"IPAddress" : "172.17.0.2",
"IPPrefixLen" : 16,
"IPv6Gateway" : "",
"GlobalIPv6Address" : "",
"GlobalIPv6PrefixLen" : 0,
"MacAddress" : "02:42:ac:11:00:02",
"DriverOpts" : null
}
}
},
"MountLabel" : "",
"State" : {
"Status" : "running",
"Running" : true,
"Paused" : false,
"Restarting" : false,
"OOMKilled" : false,
"Dead" : false,
"Pid" : 873,
"ExitCode" : 0,
"Error" : "",
"StartedAt" : "2021-10-20T20:26:53.468541843Z",
"FinishedAt" : "0001-01-01T00:00:00Z"
},
"ResolvConfPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/resolv.conf",
"ExecIDs" : null,
"HostConfig" : {
"Binds" : [ ],
"ContainerIDFile" : "",
"LogConfig" : {
"Type" : "json-file",
"Config" : { }
},
"NetworkMode" : "default",
"PortBindings" : {
"9200/tcp" : [ {
"HostIp" : "",
"HostPort" : ""
} ],
"9300/tcp" : [ {
"HostIp" : "",
"HostPort" : ""
} ]
},
"RestartPolicy" : {
"Name" : "",
"MaximumRetryCount" : 0
},
"AutoRemove" : false,
"VolumeDriver" : "",
"VolumesFrom" : [ ],
"CapAdd" : null,
"CapDrop" : null,
"CgroupnsMode" : "host",
"Dns" : null,
"DnsOptions" : null,
"DnsSearch" : null,
"ExtraHosts" : [ ],
"GroupAdd" : null,
"IpcMode" : "shareable",
"Cgroup" : "",
"Links" : null,
"OomScoreAdj" : 0,
"PidMode" : "",
"Privileged" : false,
"PublishAllPorts" : false,
"ReadonlyRootfs" : false,
"SecurityOpt" : null,
"UTSMode" : "",
"UsernsMode" : "",
"ShmSize" : 67108864,
"Runtime" : "runc",
"ConsoleSize" : [ 0, 0 ],
"Isolation" : "",
"CpuShares" : 0,
"Memory" : 0,
"NanoCpus" : 0,
"CgroupParent" : "",
"BlkioWeight" : 0,
"BlkioWeightDevice" : null,
"BlkioDeviceReadBps" : null,
"BlkioDeviceWriteBps" : null,
"BlkioDeviceReadIOps" : null,
"BlkioDeviceWriteIOps" : null,
"CpuPeriod" : 0,
"CpuQuota" : 0,
"CpuRealtimePeriod" : 0,
"CpuRealtimeRuntime" : 0,
"CpusetCpus" : "",
"CpusetMems" : "",
"Devices" : null,
"DeviceCgroupRules" : null,
"DeviceRequests" : null,
"KernelMemory" : 0,
"KernelMemoryTCP" : 0,
"MemoryReservation" : 0,
"MemorySwap" : 0,
"MemorySwappiness" : null,
"OomKillDisable" : false,
"PidsLimit" : null,
"Ulimits" : null,
"CpuCount" : 0,
"CpuPercent" : 0,
"IOMaximumIOps" : 0,
"IOMaximumBandwidth" : 0,
"MaskedPaths" : [ "/proc/asound", "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ],
"ReadonlyPaths" : [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ]
},
"Id" : "8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457",
"LogPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457-json.log"
}
Some additional log lines which might help:
INFO org.testcontainers.containers.wait.strategy.HttpWaitStrategy - /focused_roentgen: Waiting for 120 seconds for URL: http://172.17.0.1:49154/ (where port 49154 maps to container port 9200)
Log output from the failed container:
[INFO ][o.e.e.NodeEnvironment ] [dP6fGhj] using [1] data paths, mounts [[/ (overlay)]], net usable_space [79.4gb], net total_space [242.2gb], types [overlay]
[INFO ][o.e.e.NodeEnvironment ] [dP6fGhj] heap size [990.7mb], compressed ordinary object pointers [true]
[INFO ][o.e.n.Node ] [dP6fGhj] node name derived from node ID [dP6fGhj8TES6gRpq72sI0Q]; set [node.name] to override
[INFO ][o.e.n.Node ] [dP6fGhj] version[6.7.0], pid[1], build[default/docker/8453f77/2019-03-21T15:32:29.844721Z], OS[Linux/5.4.0-1028-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]
[INFO ][o.e.n.Node ] [dP6fGhj] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-2290863965764670913, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [aggs-matrix-stats]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [analysis-common]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [ingest-common]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [ingest-geoip]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [ingest-user-agent]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [lang-expression]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [lang-mustache]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [lang-painless]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [mapper-extras]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [parent-join]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [percolator]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [rank-eval]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [reindex]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [repository-url]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [transport-netty4]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [tribe]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-ccr]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-core]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-deprecation]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-graph]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-ilm]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-logstash]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-ml]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-monitoring]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-rollup]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-security]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-sql]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-upgrade]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-watcher]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] no plugins loaded
[INFO ][o.e.x.s.a.s.FileRolesStore] [dP6fGhj] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[INFO ][o.e.x.m.p.l.CppLogMessageHandler] [dP6fGhj] [controller/88] [Main.cc@109] controller (64 bit): Version 6.7.0 (Build d74ae2ac01b10d) Copyright (c) 2019 Elasticsearch BV
[INFO ][o.e.d.DiscoveryModule ] [dP6fGhj] using discovery type [single-node] and host providers [settings]
[INFO ][o.e.n.Node ] [dP6fGhj] initialized
[INFO ][o.e.n.Node ] [dP6fGhj] starting ...
[INFO ][o.e.t.TransportService ] [dP6fGhj] publish_address {172.17.0.2:9300}, bound_addresses {0.0.0.0:9300}
[INFO ][o.e.h.n.Netty4HttpServerTransport] [dP6fGhj] publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200}
[INFO ][o.e.n.Node ] [dP6fGhj] started
[2021-10-20T20:28:53.705Z] [2021-10-20T20:27:00,528][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [dP6fGhj] Failed to clear cache for realms [[]]
[INFO ][o.e.g.GatewayService ] [dP6fGhj] recovered [0] indices into cluster_state
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.watches] for index patterns [.watches*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
[INFO ][o.e.l.LicenseService ] [dP6fGhj] license [2dcb2824-6c74-4c3a-b8ac-d06233221f44] mode [basic] - valid
When I run a similar test for postgres, it passes but that's only because the postgres testcontainer uses a log watching wait strategy and doesn't try to connect to the exposed port. If I try to actually connect to the postgres container it fails as well.
My kubernetes podspec is like this:
podSpec:
spec:
containers:
- name: main
image: <redacted, but my image is based on debian>
command:
- cat
tty: true
volumeMounts:
- mountPath: /var/run
name: docker-socket
env:
- name: TESTCONTAINERS_RYUK_DISABLED
value: true
- name: dind-daemon
image: docker:20.10.9-dind-rootless
securityContext:
privileged: true
readOnlyRootFilesystem: false
volumeMounts:
- name: docker-socket
mountPath: /run/user/1000
volumes:
- name: docker-socket
emptyDir: {}
jameslamine commented
Setting TESTCONTAINERS_HOST_OVERRIDE to localhost fixed the issue. So it's not automatically detecting that correctly. The fix worked with testcontainers 1.16.1. I didn't test it with 1.16.0.
kiview commented
Thanks for reporting @jameslamine.
Indeed, there seem to be a couple of issues around this and they might very well be connected. We have to yet investigate the root cause of this.
Thanks a lot for sharing the workaround as well, much appreciated.
kiview commented
Can you please set your Testcontainers logs to DEBUG level (see here https://www.testcontainers.org/supported_docker_environment/logging_config/) and provide all the logs of the test run?
If you would be able to create some kind of local reproducer, this would also be helpful, but I get that this might be challenging or nearly impossible.
bsideup commented
The container ip address is 172.17.0.2 but it's trying to connect to 172.17.0.1 which is the gateway. Is that normal?
this is normal, as we are not connecting to container's IP, but Docker Gateway on which all ports are exposed
This could be related to #4594, except my problem happens on 1.16.0 as well. This also might be related to #4591 although I am not using TESTCONTAINERS_HOST_OVERRIDE
I think these issues are unrelated. What happening is:
- You start your tests in a pod with Docker as a sidecar container I would assume
- Testcontainers' detects that it runs in a container. This usually means that
localhostwon't work, and the gateway IP should be used instead - The gateway IP is
172.17.0.1but your pod cannot see it, as you have multiple layers of containerization
Setting TESTCONTAINERS_HOST_OVERRIDE to localhost works because containers in k8s pods share the same network namespace. Unfortunately, there isn't much we can do in this scenario, as the setup is custom. But that's exactly why we introduced TESTCONTAINERS_HOST_OVERRIDE - to be able to override the detection in the most advanced environments.
I hope this explanation helps understanding what's going on better 👍
jameslamine commented
Thanks, that's helpful.
I am running with the testcontainers logs set to debug. Unfortunately the jenkins output is a bit of a mess and it would be a lot of work for me to clean it up and redact any info from it which I should't post publicly.
I can tell you that testcontainers detects the docker socket and detects that it's running in a container and detects that the daemon is docker-in-docker rootless. You're right that it's detecting the docker host as 172.17.0.1. If there's anything specific you're looking for in the logs let me know and I can check.
Would it be possible to add an extra step to the environment detection where it starts up a docker container and tries to connect to it at both "localhost" and the gateway IP? That would be one option for automatically detecting this sort of environment. Perhaps you could use alpine for this since you're already pulling it, or make a custom tiny container to replace alpine. I don't know how much this would slow down setup.
Another option would be to try to detect that you're running inside of kubernetes and used that to help decide what to do.
jameslamine commented
I can confirm that this shows up in the logs:
INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
INFO org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
kiview commented
Hey @jameslamine, we are currently focused on investigating the other issues you have linked (which indeed seem to be a different issue) and then we can think about how to improve this use case.
Is the TESTCONTAINERS_HOST_OVERRIDE approach a productively usable workaround for you?
jameslamine commented
Yup, TESTCONTAINERS_HOST_OVERRIDE is working great. Thanks for your help with this.
kiview commented
Great to hear, I will close this issue for now, since using TESTCONTAINERS_HOST_OVERRIDE seems to be the appropriate solution for this setup.
alpeshjikadra commented
I am also facing the similar issue with testcontainers 1.16.0, 1.16.1, 1.16.2 and 1.16.3.
With kubernetes version 1.20 and 1.21.9.And adding TESTCONTAINERS_HOST_OVERRIDE=localhost is not working
Solution work for me is :
I set testcontainers gateway ip address instead of localhost, something like
TESTCONTAINERS_HOST_OVERRIDE=172.17.0.1.