Failed to detect the IPv6 address: Failed to send HTTP(S) request to "https://[2606:4700:4700::1111]/cdn-cgi/trace"
Closed this issue · 28 comments
Heya!
I was using your container to update my ipv4 and 6 for my own domain from my pi using this template:
https://github.com/novaspirit/pi-hosted/blob/d3d7e5de51d3296568c29a0f10ee334a040df943/template/portainer-v2-arm64.json#L616
But i saw in my logs that it is having some troubles with detecting my ipv6 adres:
🌐 Detected the IPv4 address: my.correct.public.ip
🤷 The A records of "mydomain.ext" are already up to date
😞 Failed to send HTTP(S) request to "https://[2606:4700:4700::1111]/cdn-cgi/trace": Get "https://[2606:4700:4700::1111]/cdn-cgi/trace": dial tcp [2606:4700:4700::1111]:443: connect: cannot assign requested address
😞 Failed to detect the IPv6 address
⏰ Checking the IP addresses in about 5m0s . . .
Outside the container doing a curl to that adres works, also on my own machine.
But i can't go into the container to test and see what the error exactly is.
Do you have any suggestions what i might have wrong? I didn't touched the TZ variable but have set it to my local timezone to see if that might change something. But sadly this also didn't fix the error i'm having.
A popular cause is that, for some reasons, you have reached the limit of the number of simultaneous TCP connections allowed by the runtime or the kernel. The Go runtime by default should reuse existing HTTP(S) connections to avoid creating new connections, though I just learned that the size of the cache could have small default values. In any case, you could use ss --all --tcp or netstat --all --tcp or other tools to check TCP connections in the TIME_WAIT state (waiting for the other end to receive the termination ACK). Those connections are morally closed but still consume resources, and they are likely the problem.
If you see lots of TIME_WAIT connections to the Cloudflare servers (the API server, 1.1.1.1, and 2606:4700:4700::1111), I am happy to take a look and check why the supposed reusing mechanism in the Go runtime is not working, maybe enlarging the cache of alive HTTP connections eventually. If you see lots of TIME_WAIT connections but they are not related to Cloudflare, it means some other programs create and close TCP connections a bit too frequently. You could check those programs or set net.ipv4.tcp_tw_reuse to 1 to allow Linux to speed up the reusing. If there are simply lots of active connections (in the ESTAB state), it is very likely that some other programs are using lots of network resources (i.e., maintaining connections to tons of clients) and there is very little I could do. It is possible that you have to change their configurations or rewrite their network handling logic.
Please let me know if you gain more information about your situation. By the way, the timezone setting should not matter---it should only affect the parsing of the update schedule (you could say, for example, "5pm everyday" using appropriate Cron strings, and the program has to know what "5pm" refers to).
To clarify, here are the default values used by the Go 1.17.8 runtime:
var DefaultTransport RoundTripper = &Transport{
Proxy: ProxyFromEnvironment,
DialContext: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}).DialContext,
ForceAttemptHTTP2: true,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
}
const DefaultMaxIdleConnsPerHost = 2The cache size should be 100 (I incorrectly said it's 2---that is the size per host). The cache should be able to handle the traffic with ease because only a few requests are made.
Another useful thing is that you could use
docker build --target builder --tag favonia/cloudflare-ddns .to locally generate a (much larger) Docker image that would contain usual Linux tools. This way, you could inspect what's going on within the container. (PS: currently I did not set the entry point for the Docker images generated this way, so I recommend running /bin/ddns manually instead of relying on the framework you are using. Let me know if this becomes a problem---it's something easy to fix.)
A popular cause is that, for some reasons, you have reached the limit of the number of simultaneous TCP connections allowed by the runtime or the kernel. The Go runtime by default should reuse existing HTTP(S) connections to avoid creating new connections, though I just learned that the size of the cache could have small default values. In any case, you could use
ss --all --tcpornetstat --all --tcpor other tools to check TCP connections in theTIME_WAITstate (waiting for the other end to receive the termination ACK). Those connections are morally closed but still consume resources, and they are likely the problem.If you see lots of
TIME_WAITconnections to the Cloudflare servers (the API server, 1.1.1.1, and 2606:4700:4700::1111), I am happy to take a look and check why the supposed reusing mechanism in the Go runtime is not working, maybe enlarging the cache of alive HTTP connections eventually. If you see lots ofTIME_WAITconnections but they are not related to Cloudflare, it means some other programs create and close TCP connections a bit too frequently. You could check those programs or setnet.ipv4.tcp_tw_reuseto1to allow Linux to speed up the reusing. If there are simply lots of active connections (in theESTABstate), it is very likely that some other programs are using lots of network resources (i.e., maintaining connections to tons of clients) and there is very little I could do. It is possible that you have to change their configurations or rewrite their network handling logic.Please let me know if you gain more information about your situation. By the way, the timezone setting should not matter---it should only affect the parsing of the update schedule (you could say, for example, "5pm everyday" using appropriate Cron strings, and the program has to know what "5pm" refers to).
First, thank you for your quick and explanation of the various causes! I hope'd i could get into the container but b/c of it's incredible small size, it also lacks the common troubleshooting tools (which is ok!).
To clarify, here are the default values used by the Go 1.17.8 runtime:
var DefaultTransport RoundTripper = &Transport{ Proxy: ProxyFromEnvironment, DialContext: (&net.Dialer{ Timeout: 30 * time.Second, KeepAlive: 30 * time.Second, }).DialContext, ForceAttemptHTTP2: true, MaxIdleConns: 100, IdleConnTimeout: 90 * time.Second, TLSHandshakeTimeout: 10 * time.Second, ExpectContinueTimeout: 1 * time.Second, } const DefaultMaxIdleConnsPerHost = 2The cache size should be 100 (I incorrectly said it's 2---that is the size per host). The cache should be able to handle the traffic with ease because only a few requests are made.
Roger, so it can't be the problem with the TCP connections that are being made. Could it maybe be the token instead? It did change the ipv4 adres when i was testing last night.
Another useful thing is that you could use
docker build --target builder --tag favonia/cloudflare-ddns .to locally generate a (much larger) Docker image that would contain usual Linux tools. This way, you could inspect what's going on within the container. (PS: currently I did not set the entry point for the Docker images generated this way, so I recommend running
/bin/ddnsmanually instead of relying on the framework you are using. Let me know if this becomes a problem---it's something easy to fix.)
Oooh that should help me alot with diagnosing the problem :) i might not get to this today, but tommorow i will look at this for sure! Thank you so much so far!
i'm testing this on a rpi but sadly the given image isn't available for arm64/v8 i'm afraid
The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
i'm testing this on a rpi but sadly the given image isn't available for arm64/v8 i'm afraid The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Take a look at the Dockerfile---it actually supports cross-compiling. 😉 Try this to build the image for linux/arm64:
docker build --target builder --tag favonia/cloudflare-ddns --build-arg TARGETOS=linux --build-arg TARGETARCH=arm64 .Updates: you can now pull favonia/cloudflare-ddns:edge-alpine instead of building the image by yourself. Let me know if there's something I could do to help you debug. I am interested in learning what's going on.
Heya! So i noticed the error changed (withoud me changing anything afaik related to the container) to:
Failed to send HTTP(S) request to "https://[2606:4700:4700::1111]/cdn-cgi/trace": Get "https://[2606:4700:4700::1111]/cdn-cgi/trace": dial tcp [2606:4700:4700::1111]:443: connect: cannot assign requested address
I did run your test image (thanks for making that one !)
docker run -e CF_API_TOKEN=tokenthatdoesworkwithipv4testedsofar -e DOMAINS=domain.ext -d favonia/cloudflare-ddns:edge-alpine
but again, even tho it's an alpine and it did pull it down, i can't exec into it with /bin/sh weirdly enough.
OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec format error: unknown
i can start /bin/ddns and it finds the ipv4 every time, but still not the ipv6 one.
Now i tried just a standard alpine image:
docker run -dt --name test3 alpine
docker exec -it test3 sh
And what i noticed is that if i wget the 1.1.1.1 it works, but not the ipv6 one!
Doing ip a showed me that it was a ipv6-less container and i'm suspecting it might be b/c of that.
Oh, then perhaps you need to adjust your Docker settings. I recommend searching for "IPv6" within README.
Aah i see! i will contact the maintainers of the template i used to use host instead! I saw it now working after recreating it!
Thank you so much for all of your help :-) <3 have a great sunday! <3
but again, even tho it's an alpine and it did pull it down, i can't exec into it with /bin/sh weirdly enough.
This is actually due to my mistake. I will fix that.
@Macleykun Updates: I believe the edge-alpine images were fixed by #125. If you have time, I would appreciate it if you could check whether they are really fixed.
docker run -e CF_API_TOKEN=therighttoken -e DOMAINS=domain.ext -d --name test favonia/cloudflare-ddns:edge-alpine
followed by
docker exec -it test /bin/sh
still resulsts in
OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec format error: unknown
on my rpi under root.
Sorry i take that back! i though i removed the image but didn't, it is now letting me enter with sh! Sorry!
And i could also determin that indeed withoud an ipv6, you get this error which explains why :D
wget https://[2606:4700:4700::1111]/cdn-cgi/trace
Connecting to [2606:4700:4700::1111] ([2606:4700:4700::1111]:443)
wget: can't connect to remote host: Address not available
I'm reopening this, b/c while the issue can be resolved easily by giving the host adapter instead of the bridge.
The instructions to add a ipv6 subnet from docker docs doesn't suffice.
I'm still finding a good way to allow containers to reach the internet on ipv6 with the bridge adapter. So far i did find this video: https://youtu.be/FOFTEsgxro4?t=642
But need to find the time to look how he makes it, i believe it does work but haven't dived into the technical details of it.
I can't use the host adapter b/c that interferes with my grafana monitoring :P which makes CF-DDNS look like it's using alot of bandwith every now and then, even tho that's incorrect.
So, the only way afaik is to use a macvlan sadly.
However, while i can add IPv6, i break IPv4 and running the same container twice isn't really handy imo.
I think for now, i'll keep using ipv4 only, unless someone knows how to add a routeable to the internet ipv6 subnet for the bridge interface (like steps)
@Macleykun Thanks for your sharing. If you have already found some tutorial explaining how to enable the IPv6, I suggest following it through first. 😃 Personally I am using docker-compose directly, and I might not have time to look into Portainer. It would be faster to check the video than waiting for me.
Not so portainer , more so not knowing how to get a bridge to route ipv6.
Tutorial worked, but ipv4 doesn’t work then for me (so back to zero xp) no hurry tho!
If you are talking about IPv6 for Docker in general, I suggest you read through the official Docker documentation on the network. The easy instructions using the "host" network work, but it seems you do not like the result. Then you should dive into technical details to configure the network exactly as you like it to be.
Been dabbeling in it, but sadly no luck so far, will try to ask more around if someone has an idea on reddit
Alright. I am closing this issue because of the following reasons:
- The tool itself is working, and no code changes can really resolve or mitigate this issue. As far as I understand, this issue is about configuring Docker networking. There's nothing this tool can do if IPv6 is not available or configured incorrectly. The tool cannot and should not change the network setting.
- I am not aware of any easy configuration for IPv6 other than using the host network. The README can only afford to explain easy configurations that will almost always work. So, no documentation should be changed either, unless another easy configuration for IPv6 is found.
In sum, this issue is about Docker, not the tool itself, and the only change I could think of is to improve the error messages. For that, another issue seems to be more appropriate.
Heya @favonia
I did a bit of research and finnaly found a way to get a dualstack working withoud the host interface!
It comes down to this command and attaching the container to it.
Given your DHCP-server gives you the following internal subnet range:
192.168.69.0/24
2001:1C05:69:69::/64
You would need to make the following ipvlan (there's no need for a macvlan, and ipvlan l3 wouldn't make this routable.
docker network create -d ipvlan \
--subnet=192.168.69.0/24 \
--ipv6 \
--subnet=2001:1C05:69:69::/64 \
--aux-address="reserveipexample1=192.168.69.2" \
--aux-address="reserveipexample2=192.168.69.3" \
-o parent=eth0 \
dualstack
If a container is made, using --network dualstack will attach it to the container, give it an IP and IPv6.
I did notice that the first (few) tries when a container runs doesn't make IPv6 work. But after the second/third/fourth try it actually is able to run!
I did notice an error 128, but a restart did fix that. I assume something send an exit command but not sure. I will keep an eye on it for a full day (my lease normally expires in an hour up to a day for IPv6, so maybe or maybe not that might have something to do with it, but i doubt it).
What you think? Would this be beneficial to the documentation of your project? I understand if you decline.
Thank you. I can add a pointer in README to this issue.
Updates: I made the following two changes. They don't solve the problem directly but should still help.
- There is a link to this issue from README.
- Failing to detect an IPv6 address at the first time will trigger the following message unless
QUIET=true.🔧 If you are using Docker, Kubernetes, or other frameworks, IPv6 networks often require additional setups.
🔧 Read more about IPv6 networks in the README at https://github.com/favonia/cloudflare-ddns
@Macleykun Thank you again for your time to investigate this difficult issue. Let me know if you think I should do something else.
mmm, i think that's all good
QUIET=true is a env variable, so people can set that themselfs :)
i saw you refered to this issue, i think that should give people enough info to get it up and running
I do wonder still, why the IPvlan IPv6 doesn't work in the beginning... like it's actual pings that aren't going trough which is odd (no route to host i believe). But hey they dissepear themselfs so that's good :D
Heya @favonia sorry to ping you yet again about this.
After some more research as i wasn't happy with using a new network (ik ik, docker wants users to make a new network, but i prefer things to be configured as easy and as simple as possible if it achieves the same things).
Based on this project/issue: robbertkl/docker-ipv6nat#65, where i wasn't so keen on b/c using NAT for IPv6, is a bit weird. I did learn that some people who referred to this issue:
Mailu/Mailu#2272
getumbrel/umbrel#1001
So based on those changes, i tried those on my pi and it actually gives a routeable IPv6 on the default bridge!
The steps are rather simple, and can also just be copied 1:1! So perhaps you or i can rewrite the IPv6 part so users can very easily use IPv6 :).
The user has to create a file:
sudo vim /etc/docker/daemon.json
Then these lines have to be added/inserted:
{
"ipv6": true,
"experimental": true,
"ip6tables": true,
"fixed-cidr-v6": "fd00:dead:beef:6969:420::/96"
}
I would advise users to keep using the first octet (fd00) as that block is meant for this kinda thing:
https://en.wikipedia.org/wiki/IPv6_address#:~:text=Unique%20local%20addresses%5Bedit%5D
The rest doesn't really matter.
After the config change is done. They have to restart the docker deamon with: sudo systemctl restart docker.service.
This might take a little while depending on the amount of containers/cpu.
I hope you want to try this change out and see it for yourself :-) i think it's a very neat solution!
{
"ipv6": true,
"experimental": true,
"ip6tables": true,
"fixed-cidr-v6": "fd00:dead:beef:6969:420::/96"
}
@Macleykun Thanks for your time checking this. I don't understand why one should append "dead:beef:6969:420" to the prefix? Could you possibly help me understand the long prefix?
the long prefix is optional, the examples i got used dead:beef and i added a few extra numbers just to add to the funny combo.
Using fd00::/8 is also more then enough :)