Failure on arm32 machine
woncheol-kim opened this issue ยท 17 comments
Hi, I've been using the old qmcgaw/cloudflare-dns-server and today I've found that the new qmcgaw/dns is available. The new docker image works fine on most of my systems except one:
It's Synology DS416j NAS which does not officially support docker, with Marvell Armada (armhf) CPU. But docker for arm32v7 works anyway with limited functionality, and the old qmcgaw/cloudflare-dns-server works fine with two obligatory options:
--network=host
--user=root
The problem is that the new qmcgaw/dns ends up with failure with messages as below:
2021/06/26 11:00:34 INFO Unbound version: 1.13.0
2021/06/26 11:00:34 INFO Settings summary:
|--Unbound settings:
|--DNS over TLS providers:
|--Cloudflare
|--Listening port: 53
|--Access control:
|--Allowed:
|--0.0.0.0/0
|--::/0
|--Caching: enabled
|--IPv4 resolution: enabled
|--IPv6 resolution: disabled
|--Verbosity level: 1/5
|--Verbosity details level: 0/4
|--Validation log level: 0/2
|--Username:
|--Blacklisting settings:
|--Blocked categories: malicious, surveillance, ads
|--Hostnames unblocked: 1
|--Additional IP networks blocked: 13
|--Check DNS: enabled
|--Update: every 24h0m0s
2021/06/26 11:00:34 INFO using DNS address 127.0.0.1 internally
2021/06/26 11:00:34 INFO generating Unbound configuration
2021/06/26 11:00:34 INFO starting unbound
2021/06/26 11:00:34 INFO healthcheck server: listening on 127.0.0.1:9999
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:0] warning: setrlimit: Operation not permitted
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:0] warning: cannot increase max open fds from 1024 to 8266
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:0] warning: continuing with less udp ports: 477
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:0] warning: increase ulimit or decrease threads, ports in config to remove this warning
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:0] notice: init module 0: validator
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:0] notice: init module 1: iterator
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:0] info: start of service (unbound 1.13.0).
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:1] error: ssl handshake failed crypto error:0D0D90AD:asn1 encoding routines:ASN1_TIME_adj:error getting time
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:1] error: and additionally crypto error:0D0D90AD:asn1 encoding routines:ASN1_TIME_adj:error getting time
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:1] error: and additionally crypto error:0D0D90AD:asn1 encoding routines:ASN1_TIME_adj:error getting time
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:1] error: and additionally crypto error:0D0D90AD:asn1 encoding routines:ASN1_TIME_adj:error getting time
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:1] error: and additionally crypto error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:1] notice: ssl handshake failed 1.0.0.1 port 853
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:1] error: ssl handshake failed crypto error:0D0D90AD:asn1 encoding routines:ASN1_TIME_adj:error getting time
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:1] error: and additionally crypto error:0D0D90AD:asn1 encoding routines:ASN1_TIME_adj:error getting time
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:1] error: and additionally crypto error:0D0D90AD:asn1 encoding routines:ASN1_TIME_adj:error getting time
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:1] error: and additionally crypto error:0D0D90AD:asn1 encoding routines:ASN1_TIME_adj:error getting time
2021/06/26 11:00:34 INFO [8573204612050321408] unbound[13:1] error: and additionally crypto error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
2021/06/26 11:00:34 WARN Caught OS signal, shutting down
2021/06/26 11:00:34 WARN healthcheck server: shutting down (context canceled)
2021/06/26 11:00:34 WARN healthcheck server: shut down
2021/06/26 11:00:39 WARN Shutdown timed out
In fact, the ssl handshake failure is repeated endlessly and the last part of the messages are the result of my interruption with ctrl+c.
See https://github.com/qdm12/gluetun/wiki/32-bit-setup
Alternatively use qmcgaw/dns:v2.0.0-beta
I'll close the issue once I add documentation for it, thanks!
I've tried qmcgaw/dns:v2.0.0-beta and it ended up with error:
docker: Error response from daemon: linux spec user: unable to find user root: no matching entries in passwd file.
Ah maybe remove the --user
flag you pass to your Docker command? The container runs on Scratch (no OS) with v2 so there is no user names. Alternatively you can use user ids. Note that you shouldn't run containers as root, ideally leave it to the default user ID 1000 (so don't specify any user).
I had already tried that and the error messages look like this:
2021/06/27 04:19:08 INFO using DNS address 127.0.0.1 internally
2021/06/27 04:19:08 INFO healthcheck server: listening on 127.0.0.1:9999
2021/06/27 04:19:08 INFO starting DNS server
2021/06/27 04:19:08 INFO DNS server listening on :53
2021/06/27 04:19:14 INFO unbound loop exited
2021/06/27 04:19:14 WARN healthcheck server: shutting down (context canceled)
2021/06/27 04:19:14 ERROR DNS server shutdown error: dns: server not started
2021/06/27 04:19:14 ERROR DNS is not working after 10 tries: lookup github.com on 1.1.1.1:53: read udp 127.0.0.1:53983->127.0.0.1:53: read: connection refused
2021/06/27 04:19:14 INFO Shutdown successful
Maybe it's time to abandon my old NAS... (frustrated)
Please don't ๐ You're helping me fix my program for your architecture, it's pretty useful, if that can motivate you!
Now these logs are from the latest or v1.x.x image, are you sure you are running that qmcgaw/dns:v2.0.0-beta
image?
If you can please create another issue if the openssl error (due to the 32 bit) is fixed now, that would be great, thanks!
edit: also going to bed will take a look tomorrow!
Thanks that you're willing to help me. :)
I don't understand your point when you mentioned openssl, but the entire messages with my docker command is as below:
docker run --rm --network=host --name cloudflare0 qmcgaw/dns:v2.0.0-beta
=========================================
========= DNS over TLS container ========
=========================================
=========================================
=== Made with โค๏ธ by github.com/qdm12 ====
=========================================
Running version v2.0.0-beta built on 2021-05-15T17:25:05Z (commit 83c928e)
๐ง Need help? https://github.com/qdm12/dns/issues/new
๐ป Email? quentin.mcgaw@gmail.com
โ Slack? Join from the Slack button on Github
๐ธ Help me? https://github.com/sponsors/qdm12
2021/06/27 04:32:02 WARN Environment variable "IPV6" is deprecated, use the following instead: DOT_CONNECT_IPV6
2021/06/27 04:32:02 WARN Environment variable "IPV4" is deprecated, use the following instead: DOT_CONNECT_IPV6
2021/06/27 04:32:02 INFO Settings summary:
|--Upstream type: DoT
|--DoT settings:
|--Resolver:
|--DNS over TLS providers:
|--Cloudflare
|--Google
|--Fallback plaintext DNS providers:
|--Cloudflare
|--Query timeout: 3s
|--Connecting over: IPv4
|--Listening port: 53
|--Caching:
|--Type: lru
|--Max entries: 10000
|--Blacklist:
|--Blacklisting is disabled
|--Blacklist settings:
|--Blocked categories: malicious
|--Additional IP networks blocked: 13
|--Check DNS: enabled
|--Update: every 24h0m0s
|--Log level: INFO
|--Query log settings:
|--Status: disabled
2021/06/27 04:32:02 INFO healthcheck server: listening on 127.0.0.1:9999
2021/06/27 04:32:02 INFO using DNS address 127.0.0.1 internally
2021/06/27 04:32:02 INFO starting DNS server
2021/06/27 04:32:02 INFO DNS server listening on :53
2021/06/27 04:32:07 INFO unbound loop exited
2021/06/27 04:32:07 WARN healthcheck server: shutting down (context canceled)
2021/06/27 04:32:07 ERROR DNS server shutdown error: dns: server not started
2021/06/27 04:32:07 ERROR DNS is not working after 10 tries: lookup github.com on 1.1.1.1:53: read udp 127.0.0.1:50010->127.0.0.1:53: read: connection refused
2021/06/27 04:32:07 INFO Shutdown successful
Now these logs are from the latest or v1.x.x image, are you sure you are running that qmcgaw/dns:v2.0.0-beta image?
I changed in 904c5d0 the unbound loop
name to run loop
since there is no more unbound
, that's why it confused me on my phone thinking you were still running v1.x.x sorry!
I don't understand your point when you mentioned openssl
unbound[13:1] error: and additionally crypto error:0D0D90AD:asn1 encoding routines:ASN1_TIME_adj:error getting time
to fix this use what's documented here; But also I recommended the alternative to use :v2.0.0-beta
so never mind, my bad! ๐
with my docker command
I think --network=host
is at fault. You should really not run this as host, there is not much point to do it anyway ๐ First try with:
docker run -it --rm --name cloudflare0 -p 53:53/udp qmcgaw/dns:v2.0.0-beta
And see if it crashes as well?
and the error messages look like this
Also for your own information, the log lines order is not intuitive, since everything in the code is quite asynchronous/parallel. From cause to effect it is actually:
2021/06/27 04:32:07 ERROR DNS is not working after 10 tries: lookup github.com on 1.1.1.1:53: read udp 127.0.0.1:50010->127.0.0.1:53: read: connection refused
2021/06/27 04:32:07 ERROR DNS server shutdown error: dns: server not started
2021/06/27 04:32:07 INFO unbound loop exited
2021/06/27 04:32:07 WARN healthcheck server: shutting down (context canceled)
2021/06/27 04:32:07 INFO Shutdown successful
So really the DNS server gets shutdown because the try to resolve github.com did not work out. For some reason it seems the DNS did not manage to start, not sure why though.
I think
--network=host
is at fault. You should really not run this as host, there is not much point to do it anyway ๐ First try with:docker run -it --rm --name cloudflare0 -p 53:53/udp qmcgaw/dns:v2.0.0-beta
Bridge network does not work on my old system. Interactive terminal for docker containers does not work either. It's kernel limitations, according to the link below:
https://stackoverflow.com/questions/52520008/can-i-install-docker-on-arm8-based-synology-nas
Now I'm really thinking of installing linux on my old i386 laptop and replace the arm32 NAS.
Now I'm really thinking of installing linux on my old i386 laptop and replace the arm32 NAS.
Let's continue the discussion on #80 for that.
For the error you see, my feeling is that there is already something listening on port 53 so the container fails to listen on it. Although it should log out something about it, I'll investigate why.
Anyway, you can try finding if something listens on port 53 with for example lsof -i -P -n | grep LISTEN
For the error you see, my feeling is that there is already something listening on port 53 so the container fails to listen on it. Although it should log out something about it, I'll investigate why.
Anyway, you can try finding if something listens on port 53 with for example
lsof -i -P -n | grep LISTEN
My system does not support lsof command, but the old version qmcgaw/cloudflare-dns-server works fine on port 53. Of course I closed the old version before I try the new one.
It should work with qmcgaw/dns:v1.5.0
as well since it uses alpine 3.12 and since you can't upgrade Docker/libseccomp2 on your host. I'll investigate why v2 doesn't work and get back to you. I will also build you a special image based on Alpine 3.12 too with the latest v1 version.
Thank you. v1.5.0 ends up saying that it does not have permission to open port 53.
Then I've tried with the option --user=root and it seems to work, but it actually partly works: sometimes (or many times) it does not reply to a domain name request.
The messages are as below:
2021/06/28 14:04:40 INFO using DNS address 127.0.0.1 internally
2021/06/28 14:04:40 INFO generating Unbound configuration
2021/06/28 14:04:40 INFO starting unbound
2021/06/28 14:04:40 INFO healthcheck server: listening on 127.0.0.1:9999
2021/06/28 14:04:40 INFO [1624889080] unbound[13:0] warning: setrlimit: Operation not permitted
2021/06/28 14:04:40 INFO [1624889080] unbound[13:0] warning: cannot increase max open fds from 1024 to 8266
2021/06/28 14:04:40 INFO [1624889080] unbound[13:0] warning: continuing with less udp ports: 477
2021/06/28 14:04:40 INFO [1624889080] unbound[13:0] warning: increase ulimit or decrease threads, ports in config to remove this warning
2021/06/28 14:04:40 INFO [1624889080] unbound[13:0] notice: init module 0: validator
2021/06/28 14:04:40 INFO [1624889080] unbound[13:0] notice: init module 1: iterator
2021/06/28 14:04:40 INFO [1624889080] unbound[13:0] info: start of service (unbound 1.10.1).
2021/06/28 14:04:40 INFO [1624889080] unbound[13:0] info: generate keytag query _ta-4a5c-4f66. NULL IN
2021/06/28 14:04:40 INFO restarting Unbound the first time to get updated files
2021/06/28 14:04:40 INFO downloading DNSSEC root hints and named root
2021/06/28 14:04:45 INFO downloading and building DNS block lists
2021/06/28 14:04:47 INFO 350398 hostnames blocked overall
2021/06/28 14:04:47 INFO 402 IP addresses blocked overall
2021/06/28 14:04:47 INFO 2490 IP networks blocked overall
2021/06/28 14:04:47 INFO generating Unbound configuration
2021/06/28 14:04:47 INFO starting unbound
2021/06/28 14:04:48 INFO [1624889088] unbound[17:0] warning: setrlimit: Operation not permitted
2021/06/28 14:04:48 INFO [1624889088] unbound[17:0] warning: cannot increase max open fds from 1024 to 8266
2021/06/28 14:04:48 INFO [1624889088] unbound[17:0] warning: continuing with less udp ports: 477
2021/06/28 14:04:48 INFO [1624889088] unbound[17:0] warning: increase ulimit or decrease threads, ports in config to remove this warning
2021/06/28 14:04:52 INFO [1624889092] unbound[17:0] notice: init module 0: validator
2021/06/28 14:04:52 INFO [1624889092] unbound[17:0] notice: init module 1: iterator
2021/06/28 14:04:52 INFO [1624889092] unbound[17:0] info: start of service (unbound 1.10.1).
2021/06/28 14:04:52 INFO [1624889092] unbound[17:1] info: generate keytag query _ta-4a5c-4f66. NULL IN
Do you see in the logs what's the Unbound version for
qmcgaw/dns:v1.5.0
- your older
qmcgaw/cloudflare-dns-server
image
I see quite a few warnings that I have not seen like setrlimit: Operation not permitted
or cannot increase max open fds from 1024 to 8266
๐ค
Anyway, give me a few hours, I'll see why v2
doesn't work in your situation. v2 does not depend on anything really, it's just a single binary without even an operating system so it should be working better anyway on corner cases hosts like yours.
Please find below the messages from qmcgaw/cloudflare-dns-server. Surprisingly, it's almost the same: the same unbound 1.10.1, the same 'setrlimit: Operation not permitted' error, 'cannot increase max open fds from 1024 to 8266' error:
2021-06-28T21:25:05.090Z INFO healthcheck server: listening on 127.0.0.1:9999
2021-06-28T21:25:05.091Z INFO using DNS address 127.0.0.1 internally
2021-06-28T21:25:05.092Z INFO generating Unbound configuration
2021-06-28T21:25:05.093Z INFO starting unbound
2021-06-28T21:25:05.109Z INFO [1624915505] unbound[11:0] warning: setrlimit: Operation not permitted
2021-06-28T21:25:05.109Z INFO [1624915505] unbound[11:0] warning: cannot increase max open fds from 1024 to 8266
2021-06-28T21:25:05.109Z INFO [1624915505] unbound[11:0] warning: continuing with less udp ports: 477
2021-06-28T21:25:05.110Z INFO [1624915505] unbound[11:0] warning: increase ulimit or decrease threads, ports in config to remove this warning
2021-06-28T21:25:05.158Z INFO [1624915505] unbound[11:0] notice: init module 0: validator
2021-06-28T21:25:05.170Z INFO [1624915505] unbound[11:0] notice: init module 1: iterator
2021-06-28T21:25:05.207Z INFO [1624915505] unbound[11:0] info: start of service (unbound 1.10.1).
2021-06-28T21:25:05.463Z INFO [1624915505] unbound[11:0] info: generate keytag query _ta-4a5c-4f66. NULL IN
2021-06-28T21:25:05.464Z INFO [1624915505] unbound[11:1] info: generate keytag query _ta-4a5c-4f66. NULL IN
2021-06-28T21:25:05.716Z INFO restarting Unbound the first time to get updated files
2021-06-28T21:25:05.716Z INFO downloading DNSSEC root hints and named root
2021-06-28T21:25:11.502Z INFO downloading and building DNS block lists
2021-06-28T21:25:14.837Z INFO 350398 hostnames blocked overall
2021-06-28T21:25:14.838Z INFO 2892 IP addresses blocked overall
2021-06-28T21:25:14.838Z INFO generating Unbound configuration
2021-06-28T21:25:15.305Z INFO starting unbound
2021-06-28T21:25:16.698Z INFO [1624915516] unbound[18:0] warning: setrlimit: Operation not permitted
2021-06-28T21:25:16.698Z INFO [1624915516] unbound[18:0] warning: cannot increase max open fds from 1024 to 8266
2021-06-28T21:25:16.699Z INFO [1624915516] unbound[18:0] warning: continuing with less udp ports: 477
2021-06-28T21:25:16.700Z INFO [1624915516] unbound[18:0] warning: increase ulimit or decrease threads, ports in config to remove this warning
2021-06-28T21:25:19.785Z INFO [1624915519] unbound[18:0] notice: init module 0: validator
2021-06-28T21:25:19.786Z INFO [1624915519] unbound[18:0] notice: init module 1: iterator
2021-06-28T21:25:19.827Z INFO [1624915519] unbound[18:0] info: start of service (unbound 1.10.1).
2021-06-28T21:25:22.344Z INFO [1624915522] unbound[18:0] info: generate keytag query _ta-4a5c-4f66. NULL IN
Try :v1.5.1
, sorry v1.5.0
had a bug in there I think. I didn't have time to dig into why :v2.0.0-beta
doesn't work, maybe try running it with --user="0"
(as root)?
Thank you. v.1.5.1 seems to work fine.
And v.2.0.0.-beta with the option --user="0" works well without any error nor warning!
Alright nice!
It was probably the same error for :v2.0.0-beta
where it could not bind to port 53 without root on your system.
On both images, we use libcap
to set low port binding capability on the binary (unbound
for v1, and the Go program I wrote for v2) with setcap 'cap_net_bind_service=+ep' unbound
. This way the container can run by default without root (safer) and have only that capability (low port number binding).
My guess is your older Docker and/or Kernel do not like that fiddling ๐ I've added a warning to the readme advising to run it as root if you run an old Docker or kernel (5b48b7f)
Closing the issue ๐