newsnowlabs/docker-ingress-routing-daemon

How to install docker-ingress-routing-daemon?

Closed this issue · 9 comments

gonzalloe commented

Hi, I've tried setup reverse proxy in Nginx before that, but no luck. After some surfing, I found this interesting solution.

What I've done is clone the project down form this repo.
However when I tried to run docker-ingress-routing-daemon, it shows docker-ingress-routing-daemon: command not found.

What should I do to start or enable the daemon? Where should I put this folder at?

struanb commented

Hi @gonzalloe and thank you for trying DIND.

The usage instructions did assume that, having cloned the repo, you manuallly changed the permissions on docker-ingress-routing-daemon to make it executable (e.g. chmod a+x docker-ingress-routing-daemon) or will run it using bash docker-ingress-routing-daemon. I have now fixed the permissions in the repo so if you git pull, or clone afresh, you won't need to do this.

They also assumed you would install the script in your PATH (e.g. by copying it to /usr/local/bin), or are running it from within the current working directory with ./docker-ingress-routing-daemon.

You will also need to run the script with the arguments you need, as explained at https://github.com/newsnowlabs/docker-ingress-routing-daemon#running-the-daemon.

I have also added instructions on how to install the example DIND systemd unit at https://github.com/newsnowlabs/docker-ingress-routing-daemon#installing-using-systemd.

I do hope this helps.

gonzalloe commented

Thanks @struanb for detailed explanation. I got DIND successfully enabled on my swarm project. The ingress gateway IP isn't showed up anymore.

But there's another issue, I only manage to see the IP of docker_gwbridge instead of the source IP from client side.

Is there any configuration to made to solve this?

struanb commented

Hi @gonzalloe. I'm glad you've got it working, or at least running.

I can't immediately explain the issue you're now reporting now though. Could you possibly provide steps to reproduce the issue?

gonzalloe commented

I've cloned the latest patch of docker-ingress-routing-daemon to my project directory, then I copy the script to /usr/local/bin to get it running.

After that I run docker-ingress-routing-daemon --ingress-gateway-ips <Node Ingress IP List> --install command on both docker nodes (Master and Slave) and I saw some messages about deleting old rules and adding ingress_sbox iptables rules.

Docker Ingress Routing Daemon 4.1.1 starting ...
Detecting ingress network and node IP:
- Ingress subnet: 10.0.0.0/24
- This node's IP: 10.0.0.2
Cleaning up any stale load-balancer rules ...
- Deleting old rule: iptables -t nat -D POSTROUTING -d 10.0.0.0/24 -m ipvs --ipvs -j ACCEPT
- Deleting old rule: iptables -t mangle -D POSTROUTING -d 10.0.0.0/24 -j TOS --set-tos 0x02/0xff
- Deleting old rule: iptables -t raw -D PREROUTING -j CT --notrack
Enumerating load balancers from --ingress-gateway-ips 10.0.0.2 10.0.0.3
- Load balancer 10.0.0.2 will have ID 2 <=== THIS NODE
- Load balancer 10.0.0.3 will have ID 3
Installing ingress namespace iptables rules for this load balancer (ID 2):
- Adding ingress_sbox iptables nat rule: iptables -t nat -I POSTROUTING -d 10.0.0.0/24 -m ipvs --ipvs -j ACCEPT
- Adding ingress_sbox iptables mangle rule: iptables -t mangle -A POSTROUTING -d 10.0.0.0/24 -j TOS --set-tos 2/0xff
- Adding ingress_sbox connection tracking disable rule: iptables -t raw -I PREROUTING -j CT --notrack
Setting ingress_sbox namespace sysctl variables:
- Setting net.ipv4.vs.conn_reuse_mode=0 net.ipv4.vs.expire_nodest_conn=1 net.ipv4.vs.expire_quiescent_template=1
Launching docker event watcher to monitor for container launches (pgroup 3129254) ...

But when I tried to trace the IP address by adding /whatsmyip behind the URL of the webpage that hosted by my container, I get the IP address of Master node's docker_gwbridge, which is 172.17.0.1 instead of the IP from client side.

The expected result would be the public IP of client side so that it can distinguish every individual clients.

struanb commented

Hi @gonzalloe. Thanks for the logs excerpt. Some follow-up questions:

  1. It looks like you have two load balancers, as from the logs I think you are running with ingress node IPs 10.0.0.2 and 10.0.0.3. Is this correct, and is your DIND launch command docker-ingress-routing-daemon --ingress-gateway-ips 10.0.0.2,10.0.0.3 --install?
  2. How many nodes are you running in your swarm altogether?
  3. Are you running DIND on every node in the swarm?
  4. On which swarm nodes are you running your actual service containers?
  5. Have you scaled your service down and up again after launching DIND? This is necessary to install the correct rules within your service containers, unless you launch DIND with --preexisting
gonzalloe commented
  1. Yes, I've double checked the command I launched in both nodes.
  2. There're 2 nodes in my Docker Swarm.
  3. Definitely.
  4. Both nodes.
  5. Yes, after I launch the DIND on both nodes, I deploy the stack up. Then I scaled the service up.
struanb commented

Hi @gonzalloe. I've been running a test on our systems to verify correct operation, and so far things check out for me.

On your nodes, please could you locate the container IDs of your service containers (using docker ps) then run this command for each <ContainerID>, which will dump out the iptables rules, policy routing rules and custom routing table rules.

nsenter -n -t $(docker inspect -f '{{.State.Pid}}' <ContainerId>) bash -c 'set -x; iptables-save -t mangle; ip rule show; ip route show table 2; ip route show table 3'

Please could you also provide details of the DIND logs after it outputs Launching docker event watcher to monitor for container launches - which logs should look roughly like the following (one block for each replica):

2022-05-17.23:05:11.109028|netstart|17218| Detected container launch for service 'test', with ID '13ab1c2403f887920968543ed53b6d4e893ecc9c39598fefc61bf232b01cc80a' and NID '17660': ingress network interface eth0 found, so applying policy routing/firewall rules:
2022-05-17.23:05:11.114758|netstart|17218| - Adding container mangle table iptables rules
2022-05-17.23:05:11.369390|netstart|17218| - Setting container sysctl net.ipv4.conf.all.rp_filter=2 net.ipv4.conf.eth0.rp_filter=2
2022-05-17.23:05:11.383432|netstart|17218| - Adding container policy routing/firewall rules for load-balancer #2 with IP 10.0.0.2
2022-05-17.23:05:12.679949|netstart|17218| - Finished configuring launched container
struanb commented

Hi @gonzalloe. Could you let me know if you are still experiencing this issue?

struanb commented

Hi @gonzalloe. Assuming you are not still experiencing this issue or working on it, we will close this issue. However if you would like to, please feel free to reopen it.