moby/hyperkit

Docker crashes when container is making lots of DNS queries

a-chernykh opened this issue · 2 comments

a-chernykh commented

Hello,

One of my containers is making 30 DNS queries / second which are resulting in failure in development environment (the failure is expected). After a minute of running, Docker crashes. I see the following in the log file:

9/20/16 10:38:11.680 PM Docker[13074]: Docker is not responding: Get http://./info: EOF: waiting 0.5s
9/20/16 10:38:12.184 PM Docker[13074]: Docker is not responding: Get http://./info: EOF: waiting 0.5s
9/20/16 10:38:12.686 PM Docker[13074]: Docker is not responding: Get http://./info: EOF: waiting 0.5s
9/20/16 10:38:13.186 PM Docker[13074]: Docker is not responding: Get http://./info: EOF: waiting 0.5s
9/20/16 10:42:51.000 PM kernel[0]: file: table is full
9/20/16 10:42:51.045 PM Docker[13073]: Socket.Datagram.input: bind returned file table overflow
9/20/16 10:42:51.000 PM kernel[0]: file: table is full
9/20/16 10:42:51.046 PM Docker[13073]: Socket.Datagram.input: bind returned file table overflow
9/20/16 10:42:51.000 PM kernel[0]: file: table is full
9/20/16 10:42:51.077 PM Docker[13073]: Socket.Datagram.input: bind returned file table overflow
9/20/16 10:42:51.000 PM kernel[0]: file: table is full
9/20/16 10:42:51.101 PM Docker[13073]: Socket.Datagram.input: bind returned file table overflow
9/20/16 10:42:51.000 PM kernel[0]: file: table is full
9/20/16 10:42:51.109 PM Docker[13073]: Socket.Datagram.input: bind returned file table overflow
9/20/16 10:42:51.000 PM kernel[0]: file: table is full

Output of lsof on host machine (Mac):

$ ps xau|grep -i docker|awk '{ print "-p "  $2 }'|xargs lsof
...
com.docke 15037 andreychernih 1071u    IPv4 0x47027f022025ad3d        0t0      UDP *:61073
com.docke 15037 andreychernih 1072u    IPv4 0x47027f022025e495        0t0      UDP *:50644
com.docke 15037 andreychernih 1073u    IPv4 0x47027f022025aad5        0t0      UDP *:58157
com.docke 15037 andreychernih 1074u    IPv4 0x47027f022025a86d        0t0      UDP *:59711
com.docke 15037 andreychernih 1075u    IPv4 0x47027f022025e6fd        0t0      UDP *:49271
com.docke 15037 andreychernih 1076u    IPv4 0x47027f022025e965        0t0      UDP *:53894
com.docke 15037 andreychernih 1077u    IPv4 0x47027f022025ebcd        0t0      UDP *:52993
com.docke 15037 andreychernih 1078u    IPv4 0x47027f022025ee35        0t0      UDP *:58466
com.docke 15037 andreychernih 1079u    IPv4 0x47027f022025a605        0t0      UDP *:50211
com.docke 15037 andreychernih 1080u    IPv4 0x47027f022025a39d        0t0      UDP *:55463
com.docke 15037 andreychernih 1081u    IPv4 0x47027f022025a135        0t0      UDP *:50014
com.docke 15037 andreychernih 1082u    IPv4 0x47027f02202697b5        0t0      UDP *:64889
com.docke 15037 andreychernih 1083u    IPv4 0x47027f022026954d        0t0      UDP *:53156
com.docke 15037 andreychernih 1084u    IPv4 0x47027f02202692e5        0t0      UDP *:52518
com.docke 15037 andreychernih 1085u    IPv4 0x47027f0220269a1d        0t0      UDP *:55100
com.docke 15037 andreychernih 1086u    IPv4 0x47027f0220269c85        0t0      UDP *:57927
com.docke 15037 andreychernih 1087u    IPv4 0x47027f0220269eed        0t0      UDP *:52377
com.docke 15037 andreychernih 1088u    IPv4 0x47027f022026907d        0t0      UDP *:57464
com.docke 15037 andreychernih 1089u    IPv4 0x47027f0220268e15        0t0      UDP *:54893
com.docke 15037 andreychernih 1090u    IPv4 0x47027f0220268bad        0t0      UDP *:63309
com.docke 15037 andreychernih 1091u    IPv4 0x47027f022026a155        0t0      UDP *:64795
com.docke 15037 andreychernih 1092u    IPv4 0x47027f022026a3bd        0t0      UDP *:58145
com.docke 15037 andreychernih 1093u    IPv4 0x47027f02202686dd        0t0      UDP *:54076
com.docke 15037 andreychernih 1094u    IPv4 0x47027f022025cc85        0t0      UDP *:49685
com.docke 15037 andreychernih 1095u    IPv4 0x47027f022026a88d        0t0      UDP *:63341
com.docke 15037 andreychernih 1096u    IPv4 0x47027f022026aaf5        0t0      UDP *:61734
com.docke 15037 andreychernih 1097u    IPv4 0x47027f022026ad5d        0t0      UDP *:50594
com.docke 15037 andreychernih 1098u    IPv4 0x47027f022026b22d        0t0      UDP *:54337
com.docke 15037 andreychernih 1099u    IPv4 0x47027f0220268475        0t0      UDP *:55839
...

Looks like kernel is running out of open files limit.

$ netstat -na
...
udp4       0      0  *.56162                *.*
udp4       0      0  *.57800                *.*
udp4       0      0  *.55147                *.*
udp4       0      0  *.57258                *.*
udp4       0      0  *.59388                *.*
udp4       0      0  *.50489                *.*
udp4       0      0  *.51222                *.*
udp4       0      0  *.57331                *.*
udp4       0      0  *.55401                *.*
udp4       0      0  *.57428                *.*
udp4       0      0  *.50763                *.*
udp4       0      0  *.49741                *.*
udp4       0      0  *.54452                *.*
udp4       0      0  *.54194                *.*
udp4       0      0  *.49581                *.*
udp4       0      0  *.65192                *.*
udp4       0      0  *.60835                *.*
udp4       0      0  *.52061                *.*
udp4       0      0  *.61005                *.*
udp4       0      0  *.57004                *.*
udp4       0      0  *.64485                *.*
udp4       0      0  *.64104                *.*
udp4       0      0  *.53502                *.*
udp4       0      0  *.64804                *.*
udp4       0      0  *.64373                *.*
udp4       0      0  *.57235                *.*
udp4       0      0  *.64774                *.*
udp4       0      0  *.58719                *.*
udp4       0      0  *.58898                *.*
...

This can be reproduced by doing something like this inside the container:

$ while [[ true ]]; do nslookup nonexistinghost; done
ijc commented

Please file a bug against https://github.com/docker/for-mac ideally by using the "Diagnostics & Feedback" item in the Whale menu so that logs will be collected and uploaded.