mozilla/ssh_scan

EADDRUNAVAIL in docker (IPv6 not configured)

deraffe opened this issue · 2 comments

deraffe commented

I tried to run ssh_scan via docker on a server that has both IPv4 and IPv6 addresses configured, but its SSH daemon only listens on the IPv4 interface.

docker run -it --rm mozilla/ssh_scan /app/bin/ssh_scan -t foo.bar
#<Thread:0x00000cf7cafb3320@/app/lib/ssh_scan/scan_engine.rb:188 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
        16: from /app/lib/ssh_scan/scan_engine.rb:191:in `block (2 levels) in scan'
        15: from /app/lib/ssh_scan/scan_engine.rb:50:in `scan_target'
        14: from /app/lib/ssh_scan/client.rb:47:in `connect'
        13: from /usr/local/lib/ruby/2.6.0/timeout.rb:108:in `timeout'
        12: from /usr/local/lib/ruby/2.6.0/timeout.rb:33:in `catch'
        11: from /usr/local/lib/ruby/2.6.0/timeout.rb:33:in `catch'
        10: from /usr/local/lib/ruby/2.6.0/timeout.rb:33:in `block in catch'
         9: from /usr/local/lib/ruby/2.6.0/timeout.rb:93:in `block in timeout'
         8: from /app/lib/ssh_scan/client.rb:48:in `block in connect'
         7: from /usr/local/lib/ruby/2.6.0/socket.rb:631:in `tcp'
         6: from /usr/local/lib/ruby/2.6.0/socket.rb:227:in `foreach'
         5: from /usr/local/lib/ruby/2.6.0/socket.rb:227:in `each'
         4: from /usr/local/lib/ruby/2.6.0/socket.rb:641:in `block in tcp'
         3: from /usr/local/lib/ruby/2.6.0/socket.rb:137:in `connect'
         2: from /usr/local/lib/ruby/2.6.0/socket.rb:56:in `connect_internal'
         1: from /usr/local/lib/ruby/2.6.0/socket.rb:1213:in `connect_nonblock'
/usr/local/lib/ruby/2.6.0/socket.rb:1213:in `__connect_nonblock': Address not available - connect(2) for [dead:beef:c0ff:ee::bee]:22 (Errno::EADDRNOTAVAIL)
Traceback (most recent call last):
        16: from /app/lib/ssh_scan/scan_engine.rb:191:in `block (2 levels) in scan'
        15: from /app/lib/ssh_scan/scan_engine.rb:50:in `scan_target'
        14: from /app/lib/ssh_scan/client.rb:47:in `connect'
        13: from /usr/local/lib/ruby/2.6.0/timeout.rb:108:in `timeout'
        12: from /usr/local/lib/ruby/2.6.0/timeout.rb:33:in `catch'
        11: from /usr/local/lib/ruby/2.6.0/timeout.rb:33:in `catch'
        10: from /usr/local/lib/ruby/2.6.0/timeout.rb:33:in `block in catch'
         9: from /usr/local/lib/ruby/2.6.0/timeout.rb:93:in `block in timeout'
         8: from /app/lib/ssh_scan/client.rb:48:in `block in connect'
         7: from /usr/local/lib/ruby/2.6.0/socket.rb:631:in `tcp'
         6: from /usr/local/lib/ruby/2.6.0/socket.rb:227:in `foreach'
         5: from /usr/local/lib/ruby/2.6.0/socket.rb:227:in `each'
         4: from /usr/local/lib/ruby/2.6.0/socket.rb:641:in `block in tcp'
         3: from /usr/local/lib/ruby/2.6.0/socket.rb:137:in `connect'
         2: from /usr/local/lib/ruby/2.6.0/socket.rb:56:in `connect_internal'
         1: from /usr/local/lib/ruby/2.6.0/socket.rb:1213:in `connect_nonblock'
/usr/local/lib/ruby/2.6.0/socket.rb:1213:in `__connect_nonblock': Address not available - connect(2) for [dead:beef:c0ff:ee::bee]:22 (Errno::EADDRNOTAVAIL)

I though this would be the result of the connection being refused, but that should have given a ECONNREFUSED instead, which is already handled in the fallback logic.

It turns out docker containers don't have IPv6 configured by default (not even for loopback), and the EADDRNOTAVAIL is the result of the network stack not being able to allocate the local address for the connection.

In systems where IPv6 is configured correctly, this error would most probably indicate that the range of local ports is exhausted (see here), which is a serious problem.
This is why I wouldn't just include the error code in the list of those that trigger the fallback to IPv4, at least not without displaying a warning.

What do you think?

claudijd commented

@deraffe I've never really got IPv6 working well with Docker yet. That said, I'm open to guidance/recommendations/PRs even on how to do it properly. I think it's less about system IPv6 and the Docker IPv6 bridge needed.

caggle commented

Apologies in advance for the long comment.

I ran into a similar problem - I am saying similar because the error message I am getting is only slightly different.

# ssh_scan -t foo.bar -p 22
#<Thread:0x00005595d77592d0@/usr/local/bundle/gems/ssh_scan-0.0.38/lib/ssh_scan/scan_engine.rb:184 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
	16: from /usr/local/bundle/gems/ssh_scan-0.0.38/lib/ssh_scan/scan_engine.rb:187:in `block (2 levels) in scan'
	15: from /usr/local/bundle/gems/ssh_scan-0.0.38/lib/ssh_scan/scan_engine.rb:51:in `scan_target'
	14: from /usr/local/bundle/gems/ssh_scan-0.0.38/lib/ssh_scan/client.rb:47:in `connect'
	13: from /usr/local/lib/ruby/2.6.0/timeout.rb:108:in `timeout'
	12: from /usr/local/lib/ruby/2.6.0/timeout.rb:33:in `catch'
	11: from /usr/local/lib/ruby/2.6.0/timeout.rb:33:in `catch'
	10: from /usr/local/lib/ruby/2.6.0/timeout.rb:33:in `block in catch'
	 9: from /usr/local/lib/ruby/2.6.0/timeout.rb:93:in `block in timeout'
	 8: from /usr/local/bundle/gems/ssh_scan-0.0.38/lib/ssh_scan/client.rb:48:in `block in connect'
	 7: from /usr/local/lib/ruby/2.6.0/socket.rb:631:in `tcp'
	 6: from /usr/local/lib/ruby/2.6.0/socket.rb:227:in `foreach'
	 5: from /usr/local/lib/ruby/2.6.0/socket.rb:227:in `each'
	 4: from /usr/local/lib/ruby/2.6.0/socket.rb:641:in `block in tcp'
	 3: from /usr/local/lib/ruby/2.6.0/socket.rb:137:in `connect'
	 2: from /usr/local/lib/ruby/2.6.0/socket.rb:56:in `connect_internal'
	 1: from /usr/local/lib/ruby/2.6.0/socket.rb:1213:in `connect_nonblock'
/usr/local/lib/ruby/2.6.0/socket.rb:1213:in `__connect_nonblock': Cannot assign requested address - connect(2) for [2406:da1c:6aa:c000:1669:34d0:5fc8:fe9f]:22 (Errno::EADDRNOTAVAIL)

My understanding is this is a Docker issue, not necessarily an ssh_scan issue.

My use case is, I am using ssh_scan in a Docker image of my own for a personal project (it's just installed as a gem). I am using the same base image as this project (ruby), if that matters. I don't have a special network configured, just the default. In my container, I can see IPv6 addresses for my interfaces. However, the container seems to disable IPv6 by default - all "kernel" parameters related to disabling of IPv6 are turned on, i.e.:

sysctl: reading key "net.ipv6.conf.all.stable_secret"
net.ipv6.conf.all.disable_ipv6 = 1
sysctl: reading key "net.ipv6.conf.default.stable_secret"
net.ipv6.conf.default.disable_ipv6 = 1
sysctl: reading key "net.ipv6.conf.eth0.stable_secret"
net.ipv6.conf.eth0.disable_ipv6 = 1
sysctl: reading key "net.ipv6.conf.ip6tnl0.stable_secret"
net.ipv6.conf.ip6tnl0.disable_ipv6 = 1
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
net.ipv6.conf.lo.disable_ipv6 = 1

I think what's happening here is, somehow the container thinks it has an IPv6 network stack, but indeed in the "kernel" level such capability is disabled. Tools like ssh_scan, which tries to resolve a DNS name to its IPv6 address first by default, barf because of this. There are a lot of similar issues in Github and in the internet overall for such behaviour.

The generally recommended workaround/fix is to forcefully enable IPv6 support as a "kernel" parameter" while building or running the docker container. There are seemingly two ways to do this:

Using the first option appears to be a valid workaround for this:

$ docker run --sysctl net.ipv6.conf.all.disable_ipv6=0 -ti mozilla/ssh_scan /app/bin/ssh_scan -t foo.bar
[
  {
    "ssh_scan_version": "0.0.37",
    "ip": "1.1.1.1",
    "hostname": "foo.bar",
    "port": 22
     ....
  }
]

I don't think this is a fix, it's rather a workaround. But perhaps ssh_scan could attempt to resolve to an IPv4 address first before trying to resolve to an IPv6 address?

Lastly, I couldn't get the second option above (via docker-compose.yml) option as I believe that's a cleaner solution (rather than specifying it as a flag in runtime), and I have no clue why :( I'd appreciate if anyone could get it to work & let me know.