panic: runtime error: invalid memory address or nil pointer dereference

Question

panic: runtime error: invalid memory address or nil pointer dereference

Closed this issue 2 years ago · 21 comments

kevindd992002 commented 2 years ago

Prerequisites

I have checked the Wiki and Discussions and found no answer
I have searched other issues and found no duplicates
I want to report a bug and not ask a question

Operating system type

Linux, Other (please mention the version in the description)

CPU architecture

AMD64

Installation

Docker

Setup

Other (please mention in the description)

AdGuard Home version

v0.108.0-b.29

Description

What did you do?

The problem started happening on March 3, 2023, 12AM GMT+8. From that point, the only thing that happened was that the container got updated to beta 29, 20 hours ago.

Expected result

DNS server works?

Actual result

The docker container keeps on restarting with this error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x7dd0bf]

Additional information

reverting back to the beta 28 and 27 containers did not fix the issue
trying the edge tag did not fix the issue

I currently pointed all my devices to my router's (pfsense) DNS service now until this issue is fixed so I'm without ad blocking in my network.

Answer 1 · 2023-03-03T11:49:57.000Z

Hello. We cannot reproduce this with that version of the image. Could you provide a full (possibly anonymized) verbose log of a failing installation?

Answer 2 · 2023-03-03T12:05:24.000Z

Here you go:

aghlog.txt

Answer 3 · 2023-03-03T12:11:12.000Z

@kevindd992002, thanks. The error in these logs is about the configuration file change from the Edge channel that hasn't landed in the Beta channel yet. The instructions on how to roll it back to run the beta version is available in the changelog.

Could you roll it back as described and reproduce the original panic message?

Answer 4 · 2023-03-03T12:13:23.000Z

Yeah, well, I didn't even try using edge until I encountered this issue. But sure, let me roll back first and go from there.

Answer 5 · 2023-03-03T12:24:59.000Z

New verbose log after rolling back schema:

aghlog.txt

Answer 6 · 2023-03-03T12:51:12.000Z

Thanks! This is a better log, but it still lacks the panic message for some reason. Perhaps, the stderr is redirected somewhere else, like syslog or journald?

An additional step you could make to try and diagnose/mitigate the issue: seeing how the last lines in the log before a restart are about the statistics, does renaming the data/stats.db file work? Perhaps, it got corrupted somehow and that now causes panics, so starting a new statistics database could work in that case.

Answer 7 · 2023-03-03T13:16:30.000Z

Here's a snip from docker logs adguard:

goroutine 32 [running]:
github.com/ameshkov/dnscrypt/v2.(*Client).DialStamp(0xc003ede000?, {{0xc008e99520, 0xc}, {0xc000100137, 0x20, 0x3a}, {0x0, 0x0, 0x0}, {0xc0090c2240, ...}, ...})
        github.com/ameshkov/dnscrypt/v2@v2.2.5/client.go:75 +0x1df
github.com/ameshkov/dnscrypt/v2.(*Client).Dial(0xc000168240?, {0xc003ede000?, 0x7fd9683c52c8?})
        github.com/ameshkov/dnscrypt/v2@v2.2.5/client.go:52 +0x125
github.com/AdguardTeam/dnsproxy/upstream.(*dnsCrypt).resetClient(0xc0004ee3f0)
        github.com/AdguardTeam/dnsproxy@v0.48.0/upstream/upstream_dnscrypt.go:93 +0x11e
github.com/AdguardTeam/dnsproxy/upstream.(*dnsCrypt).exchangeDNSCrypt(0xc0004ee3f0, 0xc003ed2090)
        github.com/AdguardTeam/dnsproxy@v0.48.0/upstream/upstream_dnscrypt.go:67 +0x10c
github.com/AdguardTeam/dnsproxy/upstream.(*dnsCrypt).Exchange(0xc0004ee3f0, 0xc003ed2090?)
        github.com/AdguardTeam/dnsproxy@v0.48.0/upstream/upstream_dnscrypt.go:34 +0x27
github.com/AdguardTeam/dnsproxy/upstream.exchangeAsync({0x1911120, 0xc0004ee3f0}, 0xc000054794?, 0x18245f8?)
        github.com/AdguardTeam/dnsproxy@v0.48.0/upstream/parallel.go:117 +0x58
created by github.com/AdguardTeam/dnsproxy/upstream.ExchangeParallel
        github.com/AdguardTeam/dnsproxy@v0.48.0/upstream/parallel.go:45 +0x18c
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x7dd0bf]

goroutine 67 [running]:
github.com/ameshkov/dnscrypt/v2.(*Client).DialStamp(0xc0002e8000?, {{0xc004042120, 0xc}, {0xc000100137, 0x20, 0x3a}, {0x0, 0x0, 0x0}, {0xc008a8c020, ...}, ...})
        github.com/ameshkov/dnscrypt/v2@v2.2.5/client.go:75 +0x1df
github.com/ameshkov/dnscrypt/v2.(*Client).Dial(0xc0002a4090?, {0xc0002e8000?, 0x7f00ffc8cbd8?})
        github.com/ameshkov/dnscrypt/v2@v2.2.5/client.go:52 +0x125
github.com/AdguardTeam/dnsproxy/upstream.(*dnsCrypt).resetClient(0xc0003b86c0)
        github.com/AdguardTeam/dnsproxy@v0.48.0/upstream/upstream_dnscrypt.go:93 +0x11e
github.com/AdguardTeam/dnsproxy/upstream.(*dnsCrypt).exchangeDNSCrypt(0xc0003b86c0, 0xc00018e1b0)
        github.com/AdguardTeam/dnsproxy@v0.48.0/upstream/upstream_dnscrypt.go:67 +0x10c
github.com/AdguardTeam/dnsproxy/upstream.(*dnsCrypt).Exchange(0xc0003b86c0, 0xc00018e1b0?)
        github.com/AdguardTeam/dnsproxy@v0.48.0/upstream/upstream_dnscrypt.go:34 +0x27
github.com/AdguardTeam/dnsproxy/upstream.exchangeAsync({0x1911120, 0xc0003b86c0}, 0x190dbb0?, 0xc000292068?)
        github.com/AdguardTeam/dnsproxy@v0.48.0/upstream/parallel.go:117 +0x58
created by github.com/AdguardTeam/dnsproxy/upstream.ExchangeParallel
        github.com/AdguardTeam/dnsproxy@v0.48.0/upstream/parallel.go:45 +0x18c
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x7dd0bf]

goroutine 12 [running]:
github.com/ameshkov/dnscrypt/v2.(*Client).DialStamp(0xc00021c080?, {{0xc003536240, 0xc}, {0xc00024e617, 0x20, 0x3a}, {0x0, 0x0, 0x0}, {0xc004abe080, ...}, ...})
        github.com/ameshkov/dnscrypt/v2@v2.2.5/client.go:75 +0x1df
github.com/ameshkov/dnscrypt/v2.(*Client).Dial(0xc00010c1b0?, {0xc00021c080?, 0x7f1ecebd0388?})
        github.com/ameshkov/dnscrypt/v2@v2.2.5/client.go:52 +0x125
github.com/AdguardTeam/dnsproxy/upstream.(*dnsCrypt).resetClient(0xc00031e6c0)
        github.com/AdguardTeam/dnsproxy@v0.48.0/upstream/upstream_dnscrypt.go:93 +0x11e
github.com/AdguardTeam/dnsproxy/upstream.(*dnsCrypt).exchangeDNSCrypt(0xc00031e6c0, 0xc00010c3f0)
        github.com/AdguardTeam/dnsproxy@v0.48.0/upstream/upstream_dnscrypt.go:67 +0x10c
github.com/AdguardTeam/dnsproxy/upstream.(*dnsCrypt).Exchange(0xc00031e6c0, 0xc00010c3f0?)
        github.com/AdguardTeam/dnsproxy@v0.48.0/upstream/upstream_dnscrypt.go:34 +0x27
github.com/AdguardTeam/dnsproxy/upstream.exchangeAsync({0x1911120, 0xc00031e6c0}, 0x0?, 0x0?)
        github.com/AdguardTeam/dnsproxy@v0.48.0/upstream/parallel.go:117 +0x58
created by github.com/AdguardTeam/dnsproxy/upstream.ExchangeParallel
        github.com/AdguardTeam/dnsproxy@v0.48.0/upstream/parallel.go:45 +0x18c

I just tried recreating the stats.db file and same thing. It restarts right exactly at the 10 seconds mark of the container being up. So it's not a corrupted stats database.

Answer 8 · 2023-03-03T13:44:09.000Z

Thanks! It looks like the bug could be in the DNSCrypt library that we use. Does the panic go away if you disable DNSCrypt?

Answer 9 · 2023-03-03T14:41:03.000Z

Thanks! It looks like the bug could be in the DNSCrypt library that we use. Does the panic go away if you disable DNSCrypt?

How do I know if I have DNSCrypt enabled (I don't have access to the GUI)? I don't even think I enabled that. This is what I have in the yaml file:

port_dnscrypt: 0
dnscrypt_config_file: ""

Answer 10 · 2023-03-03T15:25:32.000Z

I see, thanks. We're looking into this.

Answer 11 · 2023-03-15T07:06:16.000Z

same issue happened at that time(v0.107.24/v0.107.25),
and additional information for you @ainar-g
is if you set the DNS list with sdns://, with dns stamp,
such as previous adguard DNSCrypt server,
sdns://AQIAAAAAAAAAETk0LjE0MC4xNC4xNDo1NDQzINErR_JS3PLCu_iZEIbq95zkSV2LFsigxDIuUso_OQhzIjIuZG5zY3J5cHQuZGVmYXVsdC5uczEuYWRndWFyZC5jb20
then the server down, keep crashing and restart
the current solution is just edit the config(yaml) and make all this type of sdns for # comment

Answer 12 · 2023-03-15T08:01:31.000Z

same issue happened at that time(v0.107.24/v0.107.25), and additional information for you @ainar-g is if you set the DNS list with sdns://, with dns stamp, such as previous adguard DNSCrypt server, sdns://AQIAAAAAAAAAETk0LjE0MC4xNC4xNDo1NDQzINErR_JS3PLCu_iZEIbq95zkSV2LFsigxDIuUso_OQhzIjIuZG5zY3J5cHQuZGVmYXVsdC5uczEuYWRndWFyZC5jb20 then the server down, keep crashing and restart the current solution is just edit the config(yaml) and make all this type of sdns for # comment

Are you saying this is fixed in the latest version?

Answer 13 · 2023-03-15T14:01:16.000Z

@kevindd992002, @yb92k, could any of you please check the latest version on the Edge channel, v0.108.0-a.466+9f7a582d, or later? We have a tentative fix there.

Answer 14 · 2023-03-15T14:27:14.000Z

@kevindd992002, @yb92k, could any of you please check the latest version on the Edge channel, v0.108.0-a.466+9f7a582d, or later? We have a tentative fix there.

If I do update to the Edge channel, can I easily go back to stable if the problem gets fixed?

Answer 15 · 2023-03-15T14:32:43.000Z

As long as you backup your configuration file and the data/ directory and return them back once you're done, everything should work as it did.

Answer 16 · 2023-03-15T16:09:13.000Z

Ok, the latest version in the edge channel fixed it! Does that mean the fix will be deployed to the beta channel soon?

Answer 17 · 2023-03-15T16:16:17.000Z

Thanks for testing! Yes, it should be.

Answer 18 · 2023-03-15T16:18:39.000Z

How can I monitor when it goes to the latest beta channel version so I can revert to that specific version?

Answer 19 · 2023-03-15T16:20:53.000Z

I'm not sure what you mean by monitor, but all Beta and Release builds are posted to the Releases page here on GitHub. See https://github.com/AdguardTeam/AdGuardHome/releases.

Answer 20 · 2023-03-15T16:46:18.000Z

I'm not sure what you mean by monitor, but all Beta and Release builds are posted to the Releases page here on GitHub. See https://github.com/AdguardTeam/AdGuardHome/releases.

What I mean is how can I know/be notified when this specific fix is merged into the latest version in the beta channel?

Answer 21 · 2023-03-15T16:50:06.000Z

There isn't really a “merged into beta”, since Beta builds are typically built from master. Again, there isn't a notification channel other than the releases page and the update API.

If you have further questions, please open a Discussions topic instead.