NAN on Admin Stats with Multiple Containerized Pihole Admin Consoles
ORTA74 opened this issue · 15 comments
Versions
Pi-Hole: 5.18.2
FTL: 5.25.1
Web: 5.21
Docker Tag: 2024.03.2
Platform
Browser: Firefox 125, Edge 123, Edge 124 on Wintel x64
One Browser Window -- multiple tabs, each to respective Pi-hole admin console (/admin/login.php).
Two Browser Windows -- single tab, each to respective Pi-hole admin console (/admin/login.php).
Server:
Docker on RPI -- running 2x Pi-hole containers on separate mounts and containers with individual IPs on an internal network.
Running same external host IP, but different published ports for PI-ONE and PI-TWO.
** There is an NGINX SLB running DNS on /53 fronting the actual DNS serving portion of Pi-hole, but this does not appear to affect the admin pages which are served directly on Docker host IP and ports, unless there is some kind of call back... ?
Expected behavior
Each Pi-hole admin console should show independent stats to that Pi-hole instance per browser tab.
Actual behavior / bug
Each Pi-hole admin console should show independent stats.
Instead if you login to one, and then attempt to log into the other Pi-hole instances' admin console, the first Pi-hole's fields below change immediate to "NAN" in the following display boxes:
--Total queries
--Queries blocked
--Percentage blocked
--Domains on ad-lists
Steps to reproduce
-
Run two instances of Pi-hole using docker, admin consoles published to underlying system IP on separate ports (example: 10001, 10002).
-
Login to PI-ONE (hostip:10001/admin/).
-
In same browser window, open a tab and login to PI-TWO (hostip:10002/admin/)
-
As soon as that page loads and admin user authenticates to PI-TWO, PI-ONE admin stats for the 4x fields go to NAN in the other browser window/tab.
-
Also repeatable if instead of using 1x browser window and 2x tabs, using 2x browser windows with 1x tab each.
Seems to be related to the PHP handling of sessions (maybe only checking the destination IP and not assuming its possible to run admin sessions to IP obfuscated systems on ports?) This is conjecture at this point as I'm sure more investigation needed (maybe only sessionizing the destination IP and not taking into account the port number to differentiate containerized admin consoles?)
Debug Token
to be provided for both systems
Screenshots
on FireFox
on Edge 124
Additional context
Debug Token
Seems to be related to the PHP handling of sessions ...
Can you please generate a Debug Token?
Also, it would be very helpful if you include the compose file or docker run command used to start the containers.
== Debug Tokens
PI-ONE
https://tricorder.pi-hole.net/NVsoPVmK/
PI-TWO
https://tricorder.pi-hole.net/5iN1HL1n/
Configuration:
TCP/53 + UDP/53 redirected to NGINX SLB Container on Docker Host IP
Rear side of NGINX SLB round-robin SLB to internal IP's on private /24 network (1x IP for PI-ONE and 1x IP for PI-TWO).
TCP/10001 redirected to TCP/80 on PI-ONE through Docker Host IP for admin page
TCP/10002 redirected to TCP/80 on PI-TWO through Docker Host IP for admin page
NOTE: I have not tested if this issue appears using SSL on TCP/443.
// Edit: Containers are running on an RPI using Docker 26.0 and Portainer 2.19.4 to deploy, pull the image from docker.io repo and orchestrate configuration.
tcp/10001 forwarded to tcp/80. Non-privileged mode. for PI-ONE
tcp/10002 forwarded to tcp/80. Non-privileged mode. for PI-TWO
2x separate mounts in each container ---
PI-ONE:
/etc/dnsmasq.d volume mapped to /etc/dnsmasq.d
/etc/pihole volume mapped to /etc/pihole
PI-TWO:
/etc/dnsmasq.d volume mapped to /etc/dnsmasq2.d
/etc/pihole volume mapped to /etc/pihole2
Environment variables set at initial config are WEBPASSWORD and TZ (others are populated after configuration).
Network wise, internal container network setup on /24 with IP's static assigned to rear of NGINX SLB, and front of PI-ONE and front of PI-TWO. This /24 is only within Docker.
Accessing one Pi-hole container should never interfere with the other one.
Each container has its own PHP sessions. They are independent.
Did you try without NGINX SLB?
PI-TWO:
/etc/dnsmasq2.d volume mapped to /etc/dnsmasq2.d
/etc/pihole2 volume mapped to /etc/pihole2
You meant etc/pihole2 volume mapped to /etc/pihole inside the second container, right?
It should be /etc/pihole, without the ending 2.
There is no /etc/pihole2 inside the container.
The same applies to /etc/dnsmasq2.d. It should be /etc/dnsmasq.d
Comments edited to reflect the right mapping on PI-TWO. :)
Your comment is correct -- they are mapped to independent volumes, but with the proper mount points.
New update -- same issue occurs in Edge 123.0.2420.97 and 124.0.2478.51 x64 in single window multi tab, and multi window single tab so seems not browser related.
the NAN occurs on the open admin page when there is an executed login onto whichever server /admin/ page is loaded second. ie: i can load the landing page for /admin/ and all is OK right until the webpassword is entered and login button is pressed.
Could be something in the HTTP session / JSON feeds that run in the background to update stats?
I can reproduce this when I have two Pi-holes on the same physical IP
Running two containers like so:
As they are running on my desktop machine, I then access them with 127.0.0.1:10001/admin and 127.0.0.1:10002/admin respectively. The first one is fine until I log into the second one:
I suspect it's to do with the session cookie being stored against the hostname/IP (and not taking into account the port number - not even sure if that's possible)... Observe what happens when I access the first one via localhost:10001/admin and the second with 127.0.0.1:10002/admin
Both are fine.
Note also the PHP session ID when both accessing via the same IP address:
Just for fun, I also tried this with two Pi-hole v6 containers, results are much the same, though in this case we no longer have PHP:
Some suggested workarounds while we determine if this is indeed a bug, or a configuration issue:
- Host the containers on two separate machines, and thus two separate IPs
- use macvlan and give each container it's own unique IP
- have multiple hostnames for the same IP
This isn't a bug but due to the design of how cookies are handled by browsers: You cannot have two identical sites being hosted at the same IP/hostname with all major browsers I'm aware of. It can be done with, e.g., curl or any other scripting solution where you manually control cookies, though.
If you are using Firefox, you could also use "containers" and will immediately be able to work as you expect when opening the two web interfaces in distinct containers. Otherwise, my suggestion would be offering the web interface either on two different hostnames both pointing to 127.0.0.1. Without having tested it myself, this should be enough to ensure your browser is not getting confused and one cookie killing the other.
The "proper" way would be using a reverse proxy.
Similar, although not entirely the same, results with another application - log one out, second seems OK while clicking around, but note that the cookie gets deleted and reloading the page causes a need to relogin on the second one.
I think what we are seeing here is expected behaviour due to the way cookies work.
Look at PromoFaux here! Thank you for visualizing it!
Interesting -- I may try just mapping the admin consoles to an NGINX r-prox container I have on hand instead of direct mapping TCP/80 to the underlying Docker host IP on separate port-stacks... although strictly speaking is it not possible to denote a session ID based on more than hostname/IP (or something) inside of the cookie that could be linked to a login. I wonder if this behaviour also appears on TCP/443 SSL enabled sessions.
Initially MACVLAN was setup and never had an issue (although in that case, TCP/80 and TCP-UDP/53 served on individual container IPs gave no problems -- strictly once it went into sharing an underlying IP with port differentiation did it begin this cookie-behaviour.
Happy to work around it for the time being --- but I wonder if there's a creative way to solve the cookie or session setup details or something to differentiate login attempts to port-stacked admin pages on the same underlying host IP in some future version.
Just tried with independent hostnames even mapped in a hosts file, as well as my authoritative DNS server, and no session oddity going NAN with independent host names.
So Reverse Proxy /or/ independent host names from a browser perspective fixed the issue.
I'm still considering this something where the proper fix needs to happen inside browsers to differentiate applications not by host/IP alone but also on port. It'd need to be an opt-in as it'd surely break some strange legacy applications otherwise.
The relevant standard is RFC 6265, quoting the relevant parts only:
For historical reasons, cookies contain a number of security and privacy infelicities. For example, a server can indicate that a given cookie is intended for "secure" connections, but the Secure attribute does not provide integrity in the presence of an active network attacker. Similarly, cookies for a given host are shared across all the ports on that host, even though the usual "
same-origin policy" used by web browsers isolates content retrieved via different ports.
and
Cookies do not provide isolation by port. If a cookie is readable by a service running on one port, the cookie is also readable by a service running on another port of the same server. If a cookie is writable by a service on one port, the cookie is also writable by a service running on another port of the same server. For this reason, servers SHOULD NOT both run mutually distrusting services on different ports of the same host and use cookies to store security sensitive information.
The last highlighted sentence even advises against what you've tried initially. Have, I'd say Pi-hole complies with the web standards and there is actually nothing to fix here.
But I could imagine some form of warning that you have been logged out because the session cookie was changed. This should make this a lot more obvious for users seeing there same at some point in the future.
What do you think?
edit: This turns out to be nearly impossible as there is nothing we can use to identify a particular tab with once the session got invalidated/expired/deleted.
Great discussion - and seems like the limitation is cookie technology in relation to that RFC compliance. Maybe its enough to add an entry to the documentation for containerized deployments that share an underlying IP that the admin console needs a separate DNS (or hostname) or Rprox URL to differentiate the admin pages so users don't encounter the issue.
Perhaps, but I must admit that in the 8 or so years I've been on this project, I've not heard of any similar issues. Usually when people have more than one Pi-hole install, they're separated onto different hardware at least, as the main use case would be in case of failure - it's very unusual for two machines to share the same IP (unless there is some sort of fancy failover setup going on)
In this case, its a strategy to front a single VIP for clients to consume as primary DNS to the SLB, with their secondary DNS being another physical box with the same setup behind a VIP (ultimately 4x PI containers across 2 hosts and 2 SLBs) for full redundancy and service availability through only publishing 2x VIPs for standard TCP-UDP/53 publishing via a DHCP server for clients. Just the admin portion but I think we've got that licked now with the previous workarounds. I'm surprised we don't come across this cookie issue more in containerized apps (yet).
This issue is stale because it has been open 30 days with no activity. Please comment or update this issue or it will be closed in 5 days.