ix-ai/swarm-launcher

Use of Ext_Networks & Label with traefik

tscibilia opened this issue · 17 comments

tscibilia commented

Great job on this container, looks like it solves many problems with cap_add and privileged but I'm having a bit of trouble in launching it and having it connect to an external network in a traefik stack.

I'm trying to run the binhex/arch-delugevpn container and I've added an ext_network variable for traefik with the required labels under the launch_labels variable. The container runs but I can't connect to it. My guess is I have something wrong with the ext_networks variable. Any thoughts? Should I post the yml file?

tlex commented

I've just added documentation networking: https://github.com/ix-ai/swarm-launcher/wiki/networking

Please take a look and see if it helps.

  • 👍1
tscibilia commented

Thanks, that looks extremely helpful. I’ll give it another go tonight or later this week. Much appreciated!

tscibilia commented

@tlex I had to go through your networking post and give it a try (very detailed and helpful). But the particular issue I'm running into may be of my own making (I'm not sure where to troubleshoot). It seems the particular container I spoke about may potentially have a firewall issue opening a specific port '1198' for the tun traffic when I use LAUNCH_EXT_NETWORKS.

Only when I add a LAUNCH_NETWORKS variable does the container fully startup, but it doesn't seem to attach to the traefik network I mentioned. I also did a docker inspect on the container as you showed in your wiki and only the network from LAUNCH_NETWORKS showed in the 'NetworkMode'.

I inspected the traefik network and it's definitely 'attachable'. I don't think this is an issue, I'm just not that familiar with this and don't know what setting to modify. If you have any ideas, I'd greatly appreciate it. I'm not sure if the container needs a bridge network to work but that seems to be the only difference I can tell.

tlex commented

What do you want to achieve by using traefik in the setup? If I assume correctly, then you want it in order to properly connect to Deluge/Privoxy (so, the ports 8112 and 8118). The port 1198 should be outgoing, not incoming (if I understand correctly how the container is set up).

tscibilia commented

As far as I know that sounds right. Here's a post of my config.

I want to be able to access port 8112 externally via traefik, also between containers (within the same stack), I want them to connect on port 8118 for privoxy. I had this all working in docker on a single machine, but I've upgraded and want the benefits of swarm.

tlex commented

Can you please also post the output of docker service logs autodnld_deluge (make sure to also remove from there any sensitive information)? At least, the output it says Attaching to .

tscibilia commented

Sure, here's the output of that command. i put the container log into debug mode, not sure if it helps. The lines towards the end of the log shows where the container stops...

delugevpn | Error: Nexthop has invalid gateway.

delugevpn | Tue Feb 18 17:07:29 2020 write UDP: Operation not permitted (code=1)
tlex commented

It would be interesting to try starting it manually using docker-compose. The docker-compose.yml file is the one from your output:

Click here to see it 
networks:
  autodnld_internal:
    external: true
    name: autodnld_internal
  traefik_proxy:
    external: true
    name: traefik_proxy
services:
  deluge:
    cap_add:
    - NET_ADMIN
    container_name: delugevpn
    environment:
      DEBUG: "true"
      ENABLE_PRIVOXY: "yes"
      LAN_NETWORK: 192.168.5.0/24,192.168.1.0/24
      NAME_SERVERS: 8.8.8.8,8.8.4.4
      PGID: '100'
      PUID: '1044'
      STRICT_PORT_FORWARD: "yes"
      VPN_ENABLED: "yes"
      VPN_PASS: <passwd>
      VPN_PROV: pia
      VPN_USER: <username>
    image: binhex/arch-delugevpn:latest
    labels:
      ai.ix.started-by: ix.ai/swarm-launcher
      traefik.docker.network: traefik_proxy
      traefik.enable: "true"
      traefik.http.routers.deluge.rule: Host(`FQDN`)&&PathPrefix(`/deluge`)
      traefik.http.routers.deluge.tls: "true"
      traefik.http.routers.deluge.tls.certresolver: le
      traefik.http.services.deluge.loadbalancer.server.port: '8112'
    networks:
      autodnld_internal: null
      traefik_proxy: null
    ports:
    - published: 8112
      target: 8112
    - published: 8118
      target: 8118
    - published: 58846
      target: 58846
    - published: 58849
      target: 58849
    privileged: true
    restart: "no"
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock:rw
    - /mnt/ceph/docker-data/deluge:/config:rw
    - /mnt/media/downloads:/downloads:rw
    - /etc/localtime:/etc/localtime:ro
version: '3.7'

You can then try to tweak this compose file until it starts and then report back. Unfortunately, @binhex hasn't put any examples with docker-compose, but i found binhex/arch-delugevpn#134 that seems related.

tscibilia commented

Thanks, I tried it with a simple docker-compose run command and it fails the same way. I checked out the referenced post on double networks (cause it looked promising) but there is no development in that issue yet. I also modified the compose file to just use the traefik network and it still didn't work.

I think I'll open a new issue with binhex and ask about the UDP operation error.

But as I mentioned before I have this working on a separate machine as docker-compose with one external traefik network and the only difference is that traefik network is not an overlay network, the driver is set to bridge.

EDIT:
it seems to be an overlay issue, check out this forum post
https://forums.unraid.net/topic/44109-support-binhex-delugevpn/page/179/?tab=comments#elComment_738237

tscibilia commented

I hate to bring this up again but I'm still having an issue with attaching an external network to the swarm-launcher container. I tried to separate the openvpn and deluge services into two separate containers with https://hub.docker.com/r/qmcgaw/private-internet-access/

But when I launch the revised stack and inspect the container, it's automatically creating an 'autodnld_default' network and using that instead of the specified external network. I still believe I'm likely overlooking something but I was hoping to see if you had any more ideas of what might be going wrong. Do I need to specify a network for the swarm-launcher container as well as the environment variable for external networks?

tlex commented

Can you please post the stack file and the output of docker inspect that you mention? It's important to do a docker inspect on the launched container, not on the swarm-launcher container.

tscibilia commented

Thanks for checking again...

If you notice the NetworkSettings > Network section on the inspect link, it lists 'autodnld_default' with an ip address not within what I specified in the stack.yml or the log output.

Thanks for reviewing!

tlex commented

So, from what I can see there, the swarm-launcher container is the one attached to autodnld_default, which is to be expected. This container doesn't need to do do anything except make sure that the vpn container is running.

The container that you must look at is the vpn container. That one is the one started by swarm-launcher. That is the only one that communicates over the network.

As a side-note: docker stack deploy always creates a stack_default network. All the services without an explicit network declaration are attached to it. In this context, it can be ignored, because swarm-launcher starts the services outside of docker swarm.

tscibilia commented

Ah, so sorry. I was following your steps on the networking wiki where you added the external traefik network example 2, but when I got to the last step I guess I confused the container naming.

Anyway, it seems to be working with your help, Now i need to see if i can get it all the containers talking to each other!

docker inspect vpn
docker network inspect autodnld_internal -v

tlex commented

After taking a closer look at the traefik docs, I can see only one way for traefik to work in swarm mode and to add a route to the container started by swarm-launcher: you need to define a service manually, in the traefik config file (or via CLI).

Until I get around to write some docs about it, you can try to figure it out yourself here: https://docs.traefik.io/v2.0/routing/services/

In a nutshell, you need to manually define a service that points to http://vpn:8388 (or was it 8888?).

tlex commented

Short update: I could connect to a container launched by swarm-launcher via traefik with the following settings:

  • traefik started with --providers.file.filename=/config/dynamic_conf.yml (link to docs)
  • in the traefik container, the file /config/dynamic_conf.yml mounted (I've mounted it as a swarm config, but a volume mount works too)
  • both traefik and the launched containers attached to a common network

In /config/dynamic_conf.yml I have:

http:
  routers:
    vpn:
      rule: "Host(`something.ix.ai`)"
      entrypoints:
        - https
      middlewares:
        - oauth-signin@docker
        - oauth-verify@docker
        - default-https@docker
      service: vpn
      tls:
        certResolver: default
  services:
    vpn:
      loadBalancer:
        servers:
          - url: "http://my_container:943/"

If you need to connect over https to the internal container, make sure to enable insecureSkipVerify = true in your traefik configuration file (or start traefik with --serverstransport.insecureskipverify=true).

tlex commented

Last update: I've added some clarifications about ingress networks in the wiki.