Use Traefik for Edge Agent service

Question

Use Traefik for Edge Agent service

robdyke opened this issue 4 years ago · 34 comments

Portainer communicates with the edge agent over port 8000 (as per Edge Agent Guide)

The example Traefik docker-compose does not expose :8000.

As Traefik can proxy any TCP traffic, let's use it?

Answer 1 · 2020-11-19T09:57:48.000Z

Hi. No need to expose 8000. When you set up the Edge agent, you need to change the Portainer server URL to match the Edge entry point.

Answer 2 · 2020-11-19T11:22:54.000Z

Hummmm. So I set https://domain.tld in the config by the edge agent complained that it couldn't reach ws://endpoint on :8000 although it could reach the https:// on :443

Answer 3 · 2020-11-19T11:49:31.000Z

When you add an edge agent, you need to change the Portainer server's URL to point edge.yourdomain.com. With the current configuration, with any request that came for that URL, Traefik will take and redirect to the port 800 in the container.

Take note that you need to specify one URL for portainer UI that works in port 9000 and another URL for edge.

Anyway, your proposal is valid, so, please, write in a new file to have both alternatives available.

Thank you again for your contribution is very appreciated.
Ignacio

Answer 4 · 2020-11-19T12:21:12.000Z

Thanks and will do

…

On Thu, 19 Nov 2020, 11:49 Ignacio Van Droogenbroeck, < ***@***.***> wrote: When you add an edge agent, you need to change the Portainer server's URL to point edge.yourdomain.com. With the current configuration, with any request that came for that URL, Traefik will take and redirect to the port 800 in the container. Take note that you need to specify one URL for portainer UI that works in port 9000 and another URL for edge. Anyway, your proposal is valid, so, please, write in a new file to have both alternatives available. Thank you again for your contribution is very appreciated. Ignacio — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#24 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHF5AFYSY6YK3SVAB3ZZT3SQUA5TANCNFSM4T2SRRAQ> .

Answer 5 · 2020-12-22T15:10:59.000Z

I probably have the same issue or missing something. I have deployed portainer with compose and I have a resolvable second URL (edge.mydomain.com) but edge agents are unable to connect with the following ERROR:

2020/12/22 15:04:25 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed]
2020/12/22 15:04:30 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed]
2020/12/22 15:04:35 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed]
2020/12/22 15:04:40 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed]

Edge URL is set when creating the edge endpoint as https://edge.mydomain.com and it is listening but the edge agent refuses to associate. My understanding was that proxying 8000 port with traefik on the separate URL is enough.

Answer 6 · 2020-12-22T15:12:34.000Z

Paste me the logs from Portainer && agent side @baskinsy

…

On Tue, 22 Dec 2020 at 15:11, baskinsy ***@***.***> wrote: I probably have the same issue or missing something. I have deployed portainer with compose and I have a resolvable second URL ( edge.mydomain.com) but edge agents are unable to connect with the following ERROR: 2020/12/22 15:04:25 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed] 2020/12/22 15:04:30 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed] 2020/12/22 15:04:35 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed] 2020/12/22 15:04:40 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed] Edge URL is set when creating the edge endpoint as https://edge.mydomain.com and it is listening but the edge agent refuses to associate. My understanding was that proxying 8000 port with traefik on the separate URL is enough. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#24 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHF5AAZW54P25TOJCL7TVDSWCZJFANCNFSM4T2SRRAQ> .

Answer 7 · 2020-12-22T15:21:08.000Z

The only log on edge side is the above, on portainer side logs are not displaying anything at all. I made a clean test again to verify that and the only log I can find is the above ERROR on egde agent side....

Answer 8 · 2020-12-22T15:36:03.000Z

I think the tokens are broken somehow. I get "Incorrect padding" when i try to decode the token copied from portainer here https://www.base64code.com/decode

Answer 9 · 2020-12-22T17:13:38.000Z

@baskinsy I had those errors when using the provided docker-compose. I opened :8000 direct to portainer and the agent connected. Try the compose file that proxies agent through Traefik

Answer 10 · 2020-12-22T17:30:58.000Z

Yes I read your proposal to proxy also 8000 but then is the communication secured? I would try it but that means the provided docker-compose does not work for edge agents as it seems.

Answer 11 · 2020-12-22T19:29:57.000Z

@baskinsy the ws:// on :8000 is (as I understand) encrypted

Answer 12 · 2020-12-22T20:55:07.000Z

Found the issue and was confirmed by a staff member on slack. The KEY generation is broken in my case due to I'm using a four level domain for URL (edge.staging.mydomain.com).

portainer/portainer#4647

Answer 13 · 2020-12-22T21:08:53.000Z

Glad yr sorted. Opened an issue upsteam? I often use four.level.doma.is

Answer 14 · 2020-12-22T21:54:04.000Z

Yes seems the KEY cannot be decoded when a four level URL is used. When IP is used the decode works. I'll redeploy tomorrow with a third level URL on edge traefic vhost and report back if it works.

Answer 15 · 2021-01-17T21:55:24.000Z

@baskinsy Did you get it to work with a third level URL?
When i'm trying to decode my KEY i'm getting Incorrect padding aswell, but with an IP it works.

xxxx.mydomain.com

Answer 16 · 2021-01-18T14:24:49.000Z

@hSinding Yes I had success on decoding a KEY with a third level domain but I have still issues to connect the edge agent, it is correctly added and registered but cannot be browsed. Although the KEY issue can be circumvent with a third level domain, at least for me.

Answer 17 · 2021-03-17T13:10:02.000Z

Same issue here, edge just won't connect:

2021/03/17 13:09:08 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: Get https://edge1.domain.com/api/endpoints/5/status: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]

But it seems to be a traefik misconfiguration because when I call the api endpoint manually I get 404

Answer 18 · 2021-10-13T14:55:14.000Z

With a small hack I was able to make it work through Traefik :

Portainer is setup in Traefik with :

      # Frontend
      - "traefik.enable=true"
      - "traefik.http.routers.frontend-portainer.rule=Host(`portainer.mydomain.com`)"
      - "traefik.http.services.frontend-portainer.loadbalancer.server.port=9000"
      - "traefik.http.routers.frontend-portainer.service=frontend-portainer"
      - "traefik.http.routers.frontend-portainer.tls.certresolver=lets-encrypt"

      # Edge
      - traefik.http.routers.edge-portainer.rule=Host(`edge.mydomain.com`)
      - traefik.http.services.edge-portainer.loadbalancer.server.port=8000
      - traefik.http.routers.edge-portainer.service=edge-portainer
      - traefik.http.routers.edge-portainer.tls.certresolver=lets-encrypt
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.regex=^http://(.*)
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.replacement=https://$${1}
      - traefik.http.routers.http.middlewares=edge-portainer-redirect
      - traefik.http.services.http.loadbalancer.server.port=80

You first need to decode the EDGE_KEY (using https://www.base64decode.org for example) in order to obtain the fingerprint and the endpoint ID.

You should obtain something like: https://edge.mydomain.com|edge.mydomain.com:8000|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

3 being the endpoint ID, aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00 being the server fingerprint.

I modified it like that : https://portainer.mydomain.com|https://edge.mydomain.com|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

and used https://www.base64encode.org (with URL-safe encoding enabled) to regenerate the key.
Then simply use this generated EDGE_KEY on the agent and it should work without exposing any 8000port

Answer 19 · 2021-11-13T23:39:05.000Z

@charnesp this was such a handy tip. Many thanks. Portainer-peeps - it'd be fantastic to be able to generate an EDGE_KEY from the UI that supports this configuration. At very least an additional (optional) field for the edge URL would help.

Answer 20 · 2022-04-14T04:02:20.000Z

@charnesp Thanks for the help.

I'd like to add I had to add an extra step. With the new key, it complained about a deprecated MD5 fingerprint so I had to update it with a SHA256 fingerprint. No idea why.

This really needs to be an option in the UI, or at least update the docs so it's not point edge as a separate url that doesn't work without the 8000 port open.

Answer 21 · 2022-04-26T11:58:06.000Z

Hello, I am currently trying to achieve the same: Manage an Edge Agent in a VM via Portainer behind Traefik running in WSL.

As i described here, I try to browse an Edge Agent after successful association. The request to open the tunnel connection however goes to 127.0.0.1 and the connection fails.. Any advice?

Answer 22 · 2022-09-07T11:35:50.000Z

I'm had made it before like was described at this post #24 (comment)

But agent communication is possible to solve with dns:port and entrypoint also, just needed added your port of edge into traefik ports:

...
      - 8000:8000
...

And entrypoints:

...
      - --entrypoints.edge.address=:8000   
...

And add second route for this service at portainer's side:

...
    labels:
      - traefik.enable=true
      #portainer route
      - traefik.http.services.portainer.loadbalancer.server.port=9000
      - traefik.http.routers.portainer.rule=Host(`portainer.domain.ltd`)
      - traefik.http.routers.portainer.entrypoints=websecure
      - traefik.http.routers.portainer.service=portainer
      - traefik.http.routers.portainer.tls=true
      - traefik.http.routers.portainer.tls.certresolver=yourresolver
      #edge route
      - traefik.http.services.edge.loadbalancer.server.port=8000
      - traefik.http.routers.edge.rule=Host(`portainer.domain.ltd`)
      - traefik.http.routers.edge.entrypoints=edge
      - traefik.http.routers.edge.service=edge
      - traefik.http.routers.edge.tls=true
      - traefik.http.routers.edge.tls.certresolver=yourresolver
...

So, that expected, we are have portainer at portainer.domain.ltd and edge at portainer.domain.ltd:8000

Now, to access your edge, you just need to add the port to the dns name.

Answer 23 · 2022-10-19T19:57:26.000Z

In my case had to disable the tls and certresolver to make it work.
Are the data still encrypted by usinf ws instead of wss and disabling tls ?

Answer 24 · 2022-10-31T20:03:23.000Z

Thanks @SAOPP your steps here helped me, but same as @tarmacx I had to disable tls as I was facing the below error:

2022/10/31 19:53:49 client: Connecting to ws://portainer.mydomain.com:8000
2022/10/31 19:53:49 client: Connection error: websocket: bad handshake
2022/10/31 19:53:49 client: Give up

Now, to access your edge, you just need to add the port to the dns name.

I think I didn't understand this part. Where do you add the port?

By the way, it seems that we are getting really close to the correct config template. We should edit the template on the official portainer docs

Answer 25 · 2022-11-01T21:28:19.000Z

I think I didn't understand this part. Where do you add the port?

Hi! I mean portainer.mydomain.com:8000 port ova here.

Answer 26 · 2022-12-05T22:49:02.000Z

With a small hack I was able to make it work through Traefik :

Portainer is setup in Traefik with :
      # Frontend
      - "traefik.enable=true"
      - "traefik.http.routers.frontend-portainer.rule=Host(`portainer.mydomain.com`)"
      - "traefik.http.services.frontend-portainer.loadbalancer.server.port=9000"
      - "traefik.http.routers.frontend-portainer.service=frontend-portainer"
      - "traefik.http.routers.frontend-portainer.tls.certresolver=lets-encrypt"

      # Edge
      - traefik.http.routers.edge-portainer.rule=Host(`edge.mydomain.com`)
      - traefik.http.services.edge-portainer.loadbalancer.server.port=8000
      - traefik.http.routers.edge-portainer.service=edge-portainer
      - traefik.http.routers.edge-portainer.tls.certresolver=lets-encrypt
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.regex=^http://(.*)
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.replacement=https://$${1}
      - traefik.http.routers.http.middlewares=edge-portainer-redirect
      - traefik.http.services.http.loadbalancer.server.port=80
You first need to decode the EDGE_KEY (using https://www.base64decode.org for example) in order to obtain the fingerprint and the endpoint ID.

You should obtain something like: https://edge.mydomain.com|edge.mydomain.com:8000|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

3 being the endpoint ID, aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00 being the server fingerprint.

I modified it like that : https://portainer.mydomain.com|https://edge.mydomain.com|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

and used https://www.base64encode.org (with URL-safe encoding enabled) to regenerate the key. Then simply use this generated EDGE_KEY on the agent and it should work without exposing any 8000port

I tried setting things up like this but I'm getting bad handshake on the agent machine. I can't see what I did wrong there, any hints?

Answer 27 · 2022-12-06T09:01:30.000Z

Hi,

Maybe an incorrectly generated key.

You can use the following script for generating a corrected key :

#!/usr/bin/python3
from base64 import urlsafe_b64encode, urlsafe_b64decode
import argparse

INITIAL_EDGE_URL = "portainer.mydomain.com:8000"
FINAL_EDGE_URL = "https://edge.mydomain.com"

def replace_edge_address_edge_key(edge_key, initial_edge_url = INITIAL_EDGE_URL, final_edge_url = FINAL_EDGE_URL ):
    base64_bytes = edge_key.encode('utf-8')

    message_bytes = urlsafe_b64decode(base64_bytes + b'=' * (-len(base64_bytes) % 4))
    decoded_initial_key = message_bytes.decode('utf-8')
    decoded_corrected_key = decoded_initial_key.replace(initial_edge_url, final_edge_url)
    base64_bytes = decoded_corrected_key.encode('utf-8')
    message_bytes = urlsafe_b64encode(base64_bytes)
    corrected_key = message_bytes.decode('utf-8').replace('=',"")

    return corrected_key, decoded_initial_key

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('edge_key', action='store', type=str,
                        help='Initial encoded edge key')

    parser.add_argument('--initial_edge_url', action='store', type=str,
                         default=INITIAL_EDGE_URL,
                         help='Initial edge url (default: %s)' % INITIAL_EDGE_URL)

    parser.add_argument('--final_edge_url', action='store', type=str,
                         default=FINAL_EDGE_URL,
                         help='Final edge url (default: %s)' % FINAL_EDGE_URL)

    args = parser.parse_args()

    edge_key=args.edge_key
    initial_edge_url=args.initial_edge_url
    final_edge_url=args.final_edge_url

    corrected_key, decoded_initial_key = replace_edge_address_edge_key(edge_key,initial_edge_url = initial_edge_url, final_edge_url = final_edge_url )
    print("Initial decoded key: {} \nCorrected encoded key: {}".format(decoded_initial_key, corrected_key))


if __name__ == '__main__':
    main()

Answer 28 · 2022-12-06T13:30:17.000Z

Nope, that didn't do the trick

Answer 29 · 2022-12-06T13:44:09.000Z

Did you disabled TLS for your environment, or setup SSL certificate for Portainer? As Portainer is behind Traefik, which handles the HTTPS connections, you should use plain HTTP protocol.

Answer 30 · 2022-12-06T14:15:55.000Z

I've found the issue.. I had mistyped the service name in the traefik label, so the request wouldn't ever reach portainer.
"Bad handshake" is a really misleading error though

Answer 31 · 2022-12-06T16:41:16.000Z

@Jigsaw5279 Please share the details for those who are likely to experience this in the future.

Answer 32 · 2023-01-10T10:27:11.000Z

@Jigsaw5279 / @SAOPP / @tarmacx anybody that was able to solve the bad handshake issue?

2023/01/10 10:07:01 client: Connecting to ws://portainer.mydomain.com:8000
2023/01/10 10:07:01 client: Connection error: websocket: bad handshake
2023/01/10 10:07:01 client: Give up

I followed the steps from @SAOPP:

Exposed port 8000 on my traefik container & port forwarded from my router
Added edge as additional entrypoint
Added an additional route to my portainer docker container (doublechecked the service labels)

Traefik uses letsencrypt certs. I added the edge agent with these settings:

It does work when I remove the tls and certresolver labels without changing any other labels, so I'm quite sure it's not a mistyping of service names in the traefik labels...

EDIT: Tried both with force https on/off, no changes. I have to remove the tls and certresolver labels to make this work...

Answer 33 · 2023-01-31T19:42:38.000Z

For long time I had similar problem, but error was a bit different (something with ws connection give up). Then I base64 decoded EDGE_KEY and replaced second parameter (edge server address) with portainer.example.com:443, because I have found no other way to change default 8000 port which I didn't want to use.

Error has changed to bad handshake and then commenting out labels fixed the issue. Although that means certificates won't be automatically renewed at least it works. So I'm posting full configuration (in TypeScript instead of yaml) as it might help someone.

Portainer + Portainer Edge

services[portainerConfig.serviceName] = {
    container_name: portainerConfig.serviceName,
    image: `portainer/portainer-ee:2.16.2-alpine`,

    command: `-H unix:///var/run/docker.sock`,
    labels: [
        `traefik.enable=true`,

        // portainer-http
        `traefik.http.routers.${portainerConfig.serviceName}-http.entrypoints=web`,
        `traefik.http.routers.${portainerConfig.serviceName}-http.rule=Host(\`${portainerConfig.domain}\`)`,
        `traefik.http.routers.${portainerConfig.serviceName}-http.middlewares=${traefikConfig.serviceName}-redirect-to-https`,
        `traefik.http.routers.${portainerConfig.serviceName}-http.service=${portainerConfig.serviceName}`,

        // portainer
        `traefik.http.routers.${portainerConfig.serviceName}.entrypoints=websecure`,
        `traefik.http.routers.${portainerConfig.serviceName}.rule=Host(\`${portainerConfig.domain}\`)`,
        `traefik.http.routers.${portainerConfig.serviceName}.service=${portainerConfig.serviceName}`,
        `traefik.http.routers.${portainerConfig.serviceName}.tls.certresolver=acmeresolver`,
        `traefik.http.routers.${portainerConfig.serviceName}.tls.domains[0].main=${portainerConfig.domain}`,
        `traefik.http.routers.${portainerConfig.serviceName}.tls=true`,
        `traefik.http.services.${portainerConfig.serviceName}.loadbalancer.server.port=9000`,

        // portainer-edge-http
        `traefik.http.routers.${portainerConfig.serviceName}-edge-http.entrypoints=web`,
        `traefik.http.routers.${portainerConfig.serviceName}-edge-http.rule=Host(\`${portainerConfig.edge.domain}\`)`,
        `traefik.http.routers.${portainerConfig.serviceName}-edge-http.middlewares=${traefikConfig.serviceName}-redirect-to-https`,
        `traefik.http.routers.${portainerConfig.serviceName}-edge-http.service=${portainerConfig.serviceName}-edge`,

        // portainer-edge
        `traefik.http.routers.${portainerConfig.serviceName}-edge.entrypoints=websecure`,
        `traefik.http.routers.${portainerConfig.serviceName}-edge.rule=Host(\`${portainerConfig.edge.domain}\`)`,
        `traefik.http.routers.${portainerConfig.serviceName}-edge.service=${portainerConfig.serviceName}-edge`,
        // Commenting those out fixed bad handshake issue.
        // `traefik.http.routers.${portainerConfig.serviceName}-edge.tls.certresolver=acmeresolver`,
        // `traefik.http.routers.${portainerConfig.serviceName}-edge.tls.domains[0].main=${portainerConfig.edge.domain}`,
        // `traefik.http.routers.${portainerConfig.serviceName}-edge.tls=true`,
        `traefik.http.services.${portainerConfig.serviceName}-edge.loadbalancer.server.port=8000`,
    ],
    networks: [`traefik-network`],
    restart: `unless-stopped`,
    volumes: [`/var/run/docker.sock:/var/run/docker.sock`, `${sharedConfig.docker.volumes.root}/${portainerConfig.serviceName}/data:/data`],
};

Portainer Edge Agent (different machine on the Internet)

services[portainerAgentConfig.serviceName] = {
    container_name: portainerAgentConfig.serviceName,
    image: `portainer/agent:2.16.2-alpine`,

    environment: {
        EDGE: `1`,
        EDGE_ID: `<ID>`,
        EDGE_INSECURE_POLL: `0`,
        EDGE_KEY: `<BASE64_DECODED_MODIFIED_AND_ENCODED_BACK_MANUALLY>`
    },
    labels: [`traefik.enable=false`],
    networks: [`traefik-network`],
    restart: `unless-stopped`,
    volumes: [
        `/var/run/docker.sock:/var/run/docker.sock`,
        `/var/lib/docker/volumes:/var/lib/docker/volumes`,
        `/:/host`,
        `${sharedConfig.docker.volumes.root}/${portainerAgentConfig.serviceName}/data:/data`,
    ],
};

Answer 34 · 2023-06-23T20:13:13.000Z

With a small hack I was able to make it work through Traefik :

Portainer is setup in Traefik with :
      # Frontend
      - "traefik.enable=true"
      - "traefik.http.routers.frontend-portainer.rule=Host(`portainer.mydomain.com`)"
      - "traefik.http.services.frontend-portainer.loadbalancer.server.port=9000"
      - "traefik.http.routers.frontend-portainer.service=frontend-portainer"
      - "traefik.http.routers.frontend-portainer.tls.certresolver=lets-encrypt"

      # Edge
      - traefik.http.routers.edge-portainer.rule=Host(`edge.mydomain.com`)
      - traefik.http.services.edge-portainer.loadbalancer.server.port=8000
      - traefik.http.routers.edge-portainer.service=edge-portainer
      - traefik.http.routers.edge-portainer.tls.certresolver=lets-encrypt
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.regex=^http://(.*)
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.replacement=https://$${1}
      - traefik.http.routers.http.middlewares=edge-portainer-redirect
      - traefik.http.services.http.loadbalancer.server.port=80
You first need to decode the EDGE_KEY (using https://www.base64decode.org for example) in order to obtain the fingerprint and the endpoint ID.

You should obtain something like: https://edge.mydomain.com|edge.mydomain.com:8000|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

3 being the endpoint ID, aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00 being the server fingerprint.

I modified it like that : https://portainer.mydomain.com|https://edge.mydomain.com|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

and used https://www.base64encode.org (with URL-safe encoding enabled) to regenerate the key. Then simply use this generated EDGE_KEY on the agent and it should work without exposing any 8000port

Thank you much this worked perfectly!