Container starts up and looks fine, but UDMP never connects.

Question

Container starts up and looks fine, but UDMP never connects.

Closed this issue 2 years ago · 47 comments

Hi, I tried this out for the first time tonight on a UDMP version 1.7.2.2620

I ran the container attached just to watch the output and here's what I got. It seems like it's working though not sure what the warnings mean. I have the ONT plugged into Port 9 (eth8) on the UDMP. I just never get internet. I pulled the certs off my BGW210-700 using one of the usual methods. Any thoughts would be appreciated, log below.

# podman run --privileged=true --network=host --name=wpa_supplicant-udmpro -v /mnt/data/podman/wpa_supplicant/:/etc/wpa_sup
plicant/conf/ --log-driver=k8s-file --restart=on-failure -ti pbrah/wpa_supplicant-udmpro:v1.0 -Dwired -ieth8 -c/etc/wpa_sup
plicant/conf/wpa_supplicant.conf
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
Successfully initialized wpa_supplicant
eth8: Associated with 01:80:c2:00:00:03
eth8: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
eth8: CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully
eth8: CTRL-EVENT-CONNECTED - Connection to 01:80:c2:00:00:03 completed [id=0 id_str=]

Answer 1 · 2020-07-14T06:41:30.000Z

well I know --restart=on-failure is no good, i would change that to always, however I don't think thats the source of your problems. Here is a successful log

Successfully initialized wpa_supplicant
eth8: Associated with <macaddr>
eth8: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
eth8: CTRL-EVENT-EAP-STARTED EAP authentication started
eth8: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=13
eth8: CTRL-EVENT-EAP-METHOD EAP vendor 0 method 13 (TLS) selected
eth8: CTRL-EVENT-EAP-PEER-CERT depth=2 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Root CA' hash=xxx
eth8: CTRL-EVENT-EAP-PEER-CERT depth=1 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Enhanced Services CA' hash=<redacted>
eth8: CTRL-EVENT-EAP-PEER-CERT depth=0 subject='/C=US/ST=Michigan/L=Southfield/O=ATT Services Inc/OU=OCATS/CN=boostchicken-bigpimpin.sbcglobal.net' hash=<redacted>
eth8: CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:boostchicken-bigpimpin.sbcglobal.net
eth8: CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully
eth8: CTRL-EVENT-CONNECTED - Connection to <macaddr> completed [id=0 id_str=]

It looks like yours is not actually reading the peer cert. Wanna post the output of more /mnt/data/podman/wpa_supplicant/wpa_supplicant.conf and ls /mnt/data/podman/wpa_supplicant? It's hard to help you without details of your setup @impala454

Answer 2 · 2020-07-14T13:34:33.000Z

From the /mnt/data/podman/wpa_supplicant directory:

# ls
CA_001E46-27058950451040.pem
Client_001E46-27058950451040.pem
PrivateKey_PKCS1_001E46-27058950451040.pem
wpa_supplicant.conf
# more wpa_supplicant.conf
# Generated by 802.1x Credential Extraction Tool
# Copyright (c) 2018-2019 devicelocksmith.com
# Version: 1.04 windows 386
#
# Change file names to absolute paths
eapol_version=1
ap_scan=0
fast_reauth=1
network={
        ca_cert="/etc/wpa_supplicant/conf/CA_001E46-27058950451040.pem"
        client_cert="/etc/wpa_supplicant/conf/Client_001E46-27058950451040.pem"
        eap=TLS
        eapol_flags=0
        identity="<redacted>" # Internet (ONT) interface MAC address must match this value
        key_mgmt=IEEE8021X
        phase1="allow_canned_success=1"
        private_key="/etc/wpa_supplicant/conf/PrivateKey_PKCS1_001E46-27058950451040.pem"
}

I extracted the certs and files with the extract_mfg.py script (not sure whether it's ok to post where).

Answer 3 · 2020-07-15T01:01:27.000Z

@boostchicken I just noticed you redacted the mac address in your output. Is it not the standard Ethernet bridge address thing (same as mine)?

Answer 4 · 2020-07-15T03:55:13.000Z

An interesting addition. I found another guide online (basically which copy pasted from here). It mentioned power cycling the ONT after changing connections, then starting the container. When I did that, I got the same output as you @boostchicken . However, I still never got internet. What's interesting is if I killed the docker then started it again, I got the same output as my original post. I also power cycled the UDMP as well.
See here:

# podman run --privileged=true --network=host --name=wpa_supplicant-udmpro -v /mnt/data/podman/wpa_supplicant/:/etc/wpa_suppl
icant/conf/ --log-driver=k8s-file -ti pbrah/wpa_supplicant-udmpro:v1.0 -Dwired -ieth8 -c/etc/wpa_supplicant/conf/wpa_supplica
nt.conf
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
Successfully initialized wpa_supplicant
eth8: Associated with 01:80:c2:00:00:03
eth8: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
eth8: CTRL-EVENT-EAP-FAILURE EAP authentication failed
eth8: CTRL-EVENT-EAP-STARTED EAP authentication started
eth8: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=13
eth8: CTRL-EVENT-EAP-METHOD EAP vendor 0 method 13 (TLS) selected
eth8: CTRL-EVENT-EAP-PEER-CERT depth=2 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Root CA' hash=c49db95dd254851698                     3355813d223a79b59dcac26d9bbeb2739276f0f324c389
eth8: CTRL-EVENT-EAP-PEER-CERT depth=1 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Enhanced Services CA' hash=e16e0                     3391e5ef5dfe251d826c46448407254305c06b3f742db45137345003d0d
eth8: CTRL-EVENT-EAP-PEER-CERT depth=0 subject='/C=US/ST=Michigan/L=Southfield/O=ATT Services Inc/OU=OCATS/CN=aut02rcsntx.rcs                     ntx.sbcglobal.net' hash=7dd51190b562748dccb71786bca00b2537ae452716aa442be2c908683f57c311
eth8: CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:aut02rcsntx.rcsntx.sbcglobal.net
eth8: CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully
eth8: CTRL-EVENT-CONNECTED - Connection to 01:80:c2:00:00:03 completed [id=0 id_str=]
^Ceth8: CTRL-EVENT-DISCONNECTED bssid=01:80:c2:00:00:03 reason=3 locally_generated=1
eth8: CTRL-EVENT-TERMINATING
# docker ps
CONTAINER ID  IMAGE                      COMMAND     CREATED      STATUS             PORTS  NAMES
2404949cd8ae  localhost/unifi-os:latest  /sbin/init  4 weeks ago  Up 10 minutes ago         unifi-os
# docker rm wpa_supplicant-udmpro
6df2d2fdbe2dfe3401230487e86862637d115ffa7cf93d231b22d07a22fe7132
# podman run --privileged=true --network=host --name=wpa_supplicant-udmpro -v /mnt/data/podman/wpa_supplicant/:/etc/wpa_suppl
icant/conf/ --log-driver=k8s-file -ti pbrah/wpa_supplicant-udmpro:v1.0 -Dwired -ieth8 -c/etc/wpa_supplicant/conf/wpa_supplica
nt.conf
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
Successfully initialized wpa_supplicant
eth8: Associated with 01:80:c2:00:00:03
eth8: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
eth8: CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully
eth8: CTRL-EVENT-CONNECTED - Connection to 01:80:c2:00:00:03 completed [id=0 id_str=]
^Ceth8: CTRL-EVENT-DISCONNECTED bssid=01:80:c2:00:00:03 reason=3 locally_generated=1
eth8: CTRL-EVENT-TERMINATING

Answer 5 · 2020-07-25T04:21:41.000Z

Hey did you ever resolve this? Also, are you tagging the WAN? Maybe it matters up where you are? I don't tag down in Southern California

Answer 6 · 2020-07-25T04:32:15.000Z

I still haven't, unfortunately every time I have time to work on it, the wife is watching Netflix! Can you explain what you mean by tagging the WAN? I am a linux geek and software dev but fairly new to the Ubiquiti equipment. Is it just ticking the "Use VLAN ID" box and choosing a number? Is there any other UDMP side configuration I need to double check? I just had the WAN set to IPv4 DHCP, IPv6 off, and using NextDNS for DNS servers.

Answer 7 · 2020-07-25T04:35:15.000Z

on the WAN Network interface there is an option to tag the interface. In theory it should be set to 0, but the UDM bricks anyways. If you haven't done it I wouldnt worry about it.

Have you gotten eap_proxy working? Also are you sure your cert dump is good?

Answer 8 · 2020-07-25T04:40:13.000Z

Hey I saw an issue in your command line, maybe it matters.....

podman run --privileged=true --network=host --name=wpa_supplicant-udmpro -v /mnt/data/podman/wpa_supplicant/:/etc/wpa_supplicant/conf/ --log-driver=k8s-file --restart=on-failure -ti pbrah/wpa_supplicant-udmpro:v1.0 -Dwired -ieth8 -c/etc/wpa_supplicant/conf/wpa_supplicant.conf

Should be

podman run --privileged --network=host --name=wpa_supplicant-udmpro -v /mnt/data/podman/wpa_supplicant/:/etc/wpa_supplicant/conf/ --log-driver=k8s-file --restart=always -d -ti pbrah/wpa_supplicant-udmpro:v1.0 -Dwired -ieth8 -c/etc/wpa_supplicant/conf/wpa_supplicant.conf

Answer 9 · 2020-07-25T04:45:00.000Z

I have not tried the eap_proxy option yet, is that this repo? This wpa_supplicant method seemed to be the more robust solution so that's why I was trying it first. The cert dump seemed to work correctly, the files are formatted properly, and I figured my output spam seemed to work.

Now I see your reply, yeah that privileged argument is wrong. Will try that ASAP and report back.

Answer 10 · 2020-07-25T04:46:11.000Z

You absolutely want wpa_supplicant. Also there is -d to run as a daemon which is key.

Answer 11 · 2020-07-25T04:47:45.000Z

Right, I omitted the -d while troubleshooting so I could watch the console output. Will definitely put it back in once I verify it works.

Answer 12 · 2020-07-25T05:10:39.000Z

You can just do podman logs -f for a tail like follow. No need to not run it as a daemon

Answer 13 · 2020-07-25T14:51:02.000Z

Tried again, same drill:

Successfully initialized wpa_supplicant
eth8: Associated with 01:80:c2:00:00:03
eth8: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
eth8: CTRL-EVENT-EAP-STARTED EAP authentication started
eth8: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=13
eth8: CTRL-EVENT-EAP-METHOD EAP vendor 0 method 13 (TLS) selected
eth8: CTRL-EVENT-EAP-PEER-CERT depth=2 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Root CA' hash=c49db95dd2548516983355813d223a79b59dcac26d9bbeb2739276f0f324c389
eth8: CTRL-EVENT-EAP-PEER-CERT depth=1 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Enhanced Services CA' hash=e16e03391e5ef5dfe251d826c46448407254305c06b3f742db45137345003d0d
eth8: CTRL-EVENT-EAP-PEER-CERT depth=0 subject='/C=US/ST=Michigan/L=Southfield/O=ATT Services Inc/OU=OCATS/CN=aut02rcsntx.rcsntx.sbcglobal.net' hash=7dd51190b562748dccb71786bca00b2537ae452716aa442be2c908683f57c311
eth8: CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:aut02rcsntx.rcsntx.sbcglobal.net
eth8: CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully
eth8: CTRL-EVENT-CONNECTED - Connection to 01:80:c2:00:00:03 completed [id=0 id_str=]

So it appears to work! But my UDMP never connects. ~~What's super annoying is I can't connect to the UDMP web interface if it doesn't have internet (that must be a setting somewhere because that's ridiculous)~~ (figured that out that I didn't have a local account). My eth8 shows an external IP as expected and everything, though it's the same IP it had previously through the BGW.

I'm not convinced it's not some "obvious" setting or order of operations I'm missing. My steps:

Turn off ONT
Unplug ONT Ethernet from BGW
Plug ONT Ethernet into eth8 (port 9) of UDMP
Turn ONT back on
Run docker

Does that follow with what others are doing? Also has anyone confirmed this working with version 1.7.2.2620 (the current version of UnifiOS).

Answer 14 · 2020-07-28T21:45:24.000Z

Looks good to me! I don't have to do all that with my ONT regularly. I had to do it once when it went haywire but it works just fine anytime i reboot.

Answer 15 · 2020-08-30T06:20:27.000Z

I had the exact same issue and fixed it by adding the following to the boot script... mac address should be the same as in your WPA supplicant. This will change the mac address on the wan port so it matches your uverse gateway

ip link set dev eth8 address mac-address;

I have tested this on UDM PRO 1.8.1-rc.3

Answer 16 · 2020-08-30T13:34:32.000Z

@alloylab that was totally it!! Thanks!! Which boot script exactly did you change by the way?

Answer 17 · 2020-08-30T14:22:25.000Z

Spoke too soon, I think that fixed the issue where when the docker successfully ran and authenticated I now get internet, but now on the reconnect it fails similar to my original post. It seems to authenticate with the full proper response once, then subsequent authentications give the shortened, failed response from the original post.

Answer 18 · 2020-09-03T02:14:08.000Z

@impala454 i just added a boot script based on @boostchicken udm-utilities

I have the same issue on re-connect... but how often are you re-connecting? my AT&T fiber never goes down...

Answer 19 · 2020-09-03T02:56:49.000Z

Mine rarely goes down either. I did use @boostchicken 's boot script. Maybe my script isn't great. Could you post what your boot script looks like? When I say mine doesn't auth on reconnect, I'm saying it worked for like 10-15 minutes and then I lost it. Upon losing it I checked the logs and then restarted the docker and I got the output I posted in my first post above and it never reconnected. I have a glimmer of hope though as after your suggestion to change the MAC address I saw it work for the first time.

Answer 20 · 2020-09-03T03:07:47.000Z

Here you go:

podman run --privileged --network=host --name=wpa_supplicant-udmpro -v /mnt/data/podman/wpa_supplicant/:/etc/wpa_supplicant/conf/ --log-driver=k8s-file --restart=always -d -ti pbrah/wpa_supplicant-udmpro:v1.0 -Dwired -ieth8 -c /etc/wpa_supplicant/conf/wpa_supplicant.conf;

/mnt/data/on_boot.d/10-iplink_config.sh
#!/bin/sh

ip link set dev eth8 address mac-address-redacted;

/mnt/data/on_boot.d/20-wpa_supplicant.sh
#!/bin/sh

podman start wpa_supplicant-udmpro;

Answer 21 · 2020-09-05T12:24:29.000Z

Ah, I think I see the issue I was having. I was actually creating the docker from the boot script rather than just running it. Fixed that and rebooted the UDMP and now it's working again. I'll give it a day or two before I call it 100% good and close the ticket. Thanks again @alloylab .

Answer 22 · 2020-09-05T13:08:50.000Z

Same drill as before. It works for about 20 minutes or so then the connection simply drops. I did a docker restart on it and it just didn't work. Here's the output from the two runs:

Successful run on the initial reboot:

Successfully initialized wpa_supplicant
eth8: Associated with 01:80:c2:00:00:03
eth8: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
eth8: CTRL-EVENT-EAP-STARTED EAP authentication started
eth8: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=13
eth8: CTRL-EVENT-EAP-METHOD EAP vendor 0 method 13 (TLS) selected
eth8: CTRL-EVENT-EAP-PEER-CERT depth=2 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Root CA'     hash=c49db95dd2548516983355813d223a79b59dcac26d9bbeb2739276f0f324c389
eth8: CTRL-EVENT-EAP-PEER-CERT depth=1 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Enhanced Services CA' hash=e16e03391e5ef5dfe251d826c46448407254305c06b3f742db45137345003d0d
eth8: CTRL-EVENT-EAP-PEER-CERT depth=0 subject='/C=US/ST=Michigan/L=Southfield/O=ATT Services     Inc/OU=OCATS/CN=aut02rcsntx.rcsntx.sbcglobal.net' hash=dfbcfce4dd28fa3ad96c06a79c9913c1f5edd045fd0731134f85eaa6116e7ff8
eth8: CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:aut02rcsntx.rcsntx.sbcglobal.net
eth8: CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully
eth8: CTRL-EVENT-CONNECTED - Connection to 01:80:c2:00:00:03 completed [id=0 id_str=]
eth8: CTRL-EVENT-DISCONNECTED bssid=01:80:c2:00:00:03 reason=3 locally_generated=1
eth8: CTRL-EVENT-TERMINATING

Failed run after docker restart (failed as in it never connects).

Successfully initialized wpa_supplicant
eth8: Associated with 01:80:c2:00:00:03
eth8: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
eth8: CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully
eth8: CTRL-EVENT-CONNECTED - Connection to 01:80:c2:00:00:03 completed [id=0 id_str=]

Answer 23 · 2020-09-05T13:10:17.000Z

At least there's a disconnected reason code. Looking that up it looks like:

3, "Deauthenticated because sending STA is leaving (or has left) IBSS or ESS"

Answer 24 · 2020-09-05T14:02:53.000Z

@alloylab would you be able to post your UDMP WAN config page (with redacts of course). I wonder if I have something wrong there. Mine are:

IPv4: Using DHCP
IPv6: Disabled
DNS: (my own NextDNS servers)
Use VLAN ID: Off
Report Interface Events: On
Enable Smart Queues: Off

Answer 25 · 2020-09-05T15:28:03.000Z

Only difference is my dns is 8.8.8.8 & 1.1.1.1 and I have smart queues enabled.

Answer 26 · 2020-09-05T15:33:40.000Z

alloylab commented 4 years ago

Answer 27 · 2020-09-05T15:34:54.000Z

Is your UDM-Pro still at 1.7.2?

Answer 28 · 2020-09-05T15:38:50.000Z

It's currently on 1.8.0.2888

Answer 29 · 2020-09-05T15:40:11.000Z

One thing I'm curious about, I didn't redact the MAC address from my runs because it's a plain old standard IEEE Std 802.1X PAE address. Does your authentication log show this same address, or something that actually looks unique?

Answer 30 · 2020-09-05T15:42:37.000Z

Also maybe curious if there's any diffs between my wpa_supplicant.conf and yours. Mine is posted above #9 (comment)

Answer 31 · 2020-09-05T23:45:12.000Z

eth8: Associated with 01:80:c2:00:00:03
eth8: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
eth8: CTRL-EVENT-EAP-STARTED EAP authentication started
eth8: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=13
eth8: CTRL-EVENT-EAP-METHOD EAP vendor 0 method 13 (TLS) selected
eth8: CTRL-EVENT-EAP-PEER-CERT depth=2 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Root CA' hash=**
eth8: CTRL-EVENT-EAP-PEER-CERT depth=1 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Enhanced Services CA' hash=**
eth8: CTRL-EVENT-EAP-PEER-CERT depth=0 subject='/C=US/ST=Michigan/L=Southfield/O=ATT Services Inc/OU=OCATS/CN=aut02rcsntx.rcsntx.sbcglobal.net' hash=**
eth8: CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:aut02rcsntx.rcsntx.sbcglobal.net
eth8: CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully
eth8: CTRL-EVENT-CONNECTED - Connection to 01:80:c2:00:00:03 completed [id=0 id_str=]

Answer 32 · 2020-09-05T23:47:00.000Z

eapol_version=1
ap_scan=0
fast_reauth=1
network={
ca_cert="/etc/wpa_supplicant/conf/CA.pem"
client_cert="/etc/wpa_supplicant/conf/Client.pem"
eap=TLS
eapol_flags=0
identity="MAC:Address" # Internet (ONT) interface MAC address must match this value
key_mgmt=IEEE8021X
phase1="allow_canned_success=1"
private_key="/etc/wpa_supplicant/conf/PrivateKey.pem"
}

Answer 33 · 2020-09-05T23:49:31.000Z

everything seems to match, can you send me your podman run command that you used?

Answer 34 · 2020-09-05T23:54:57.000Z

Do you have WAN2 disabled? Also, after the connection drops after 20 minutes... is the unifi-os container still running and do you see any interface events in the unifi controller?

Answer 35 · 2020-09-06T02:14:35.000Z

I actually blew away the previous podman docker and pasted your run command. I do have WAN2 disabled (it actually doesn't even exist). Should it be there? I blew it away a long time ago because I didn't think I needed it. When the connection drops everything else is still running and I can get connectivity back by simply replugging the cables back into the AT&T gateway as before.

Answer 36 · 2020-09-06T02:15:15.000Z

I also added VLAN ID 0 to the WAN connection as I'd seen it on lots of other suggestions.

Answer 37 · 2020-09-06T02:59:40.000Z

I have WAN2 disabled and I had no luck with setting VLANID to 0. Did you have any events in UniFi controller?

Answer 38 · 2020-09-06T03:44:50.000Z

Yeah I had to switch to "Classic" mode to get to set that, then realized that errored out. I changed it back and retrying the run with -d in the wpa_supplicant command for debugging this time.

Answer 39 · 2020-09-06T04:50:43.000Z

Worked again for about 30-40 minutes then just abruptly stops. No log from unifi other than to say eth8 had gone down. I might factory reset the UDMP in my next attempt.

Answer 40 · 2020-09-08T22:59:04.000Z

@impala454 any luck with doing a factory reset? Any additional log info from running in debug mode?

Answer 41 · 2020-09-08T23:31:10.000Z

I ran with debug mode and it gave lots of additional info but no smoking guns I could see. Attached is a (hopefully) redacted log of several attempts after switching to debug mode. Maybe you could see something I don't. Will try the factory reset as soon as I can anger the family with no internet for a few hours, ha.
docker_logs_redacted.txt

Answer 42 · 2020-09-08T23:43:24.000Z

@impala454 line 437 is interesting... "Request to deauthenticate"... what is requesting the deauth...

2020-09-05T23:13:46.729984273-05:00 eth8: Request to deauthenticate - bssid=01:80:c2:00:00:03 pending_bssid=00:00:00:00:00:00 reason=3 state=COMPLETED 2020-09-05T23:13:46.729984273-05:00 eth8: Event DEAUTH (11) received 2020-09-05T23:13:46.729984273-05:00 eth8: Deauthentication notification 2020-09-05T23:13:46.729984273-05:00 eth8: * reason 3 (locally generated) 2020-09-05T23:13:46.729984273-05:00 Deauthentication frame IE(s) - hexdump(len=0): [NULL] 2020-09-05T23:13:46.729984273-05:00 eth8: CTRL-EVENT-DISCONNECTED bssid=01:80:c2:00:00:03 reason=3 locally_generated=1

Answer 43 · 2020-09-08T23:52:49.000Z

Yeah it makes me wonder if this has worked at all, and that it only "worked" previously because I'm basically spoofing the MAC address of the AT&T gateway. I'll prove this theory out by stopping the docker and then swapping the cables and see if it stays up for roughly the same amount of time.

I'm also not 100% following what this setup does in its nominal config. Is it the ONT that requests authorization from the gateway? Or the gateway is supposed to periodically send auth info to the ONT?

Answer 44 · 2020-11-05T01:40:39.000Z

I think I might be having the same issue.. I've tried all the steps in here and still no luck. One thing is that I think mine keeps repeating. So mine looks like this:
eth8: CTRL-EVENT-EAP-STARTED EAP authentication started eth8: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=13 eth8: CTRL-EVENT-EAP-METHOD EAP vendor 0 method 13 (TLS) selected eth8: CTRL-EVENT-EAP-PEER-CERT depth=2 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Root CA' hash=REDACTED eth8: CTRL-EVENT-EAP-PEER-CERT depth=1 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Enhanced Services CA' hash=REDACTED eth8: CTRL-EVENT-EAP-PEER-CERT depth=0 subject='/C=US/ST=Michigan/L=Southfield/O=ATT Services Inc/OU=OCATS/CN=aut00asm.asm.bellsouth.net' hash=REDACTED eth8: CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:aut00asm.asm.bellsouth.net eth8: CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully eth8: CTRL-EVENT-EAP-STARTED EAP authentication started eth8: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=13 eth8: CTRL-EVENT-EAP-METHOD EAP vendor 0 method 13 (TLS) selected eth8: CTRL-EVENT-EAP-PEER-CERT depth=2 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Root CA' hash=REDACTED eth8: CTRL-EVENT-EAP-PEER-CERT depth=1 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Enhanced Services CA' hash=REDACTED eth8: CTRL-EVENT-EAP-PEER-CERT depth=0 subject='/C=US/ST=Michigan/L=Southfield/O=ATT Services Inc/OU=OCATS/CN=aut00asm.asm.bellsouth.net' hash=REDACTED eth8: CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:aut00asm.asm.bellsouth.net eth8: CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully eth8: CTRL-EVENT-EAP-STARTED EAP authentication started eth8: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=13 eth8: CTRL-EVENT-EAP-METHOD EAP vendor 0 method 13 (TLS) selected eth8: CTRL-EVENT-EAP-PEER-CERT depth=2 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Root CA' hash=REDACTED eth8: CTRL-EVENT-EAP-PEER-CERT depth=1 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Enhanced Services CA' hash=REDACTED eth8: CTRL-EVENT-EAP-PEER-CERT depth=0 subject='/C=US/ST=Michigan/L=Southfield/O=ATT Services Inc/OU=OCATS/CN=aut00asm.asm.bellsouth.net' hash=REDACTED eth8: CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:aut00asm.asm.bellsouth.net eth8: CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully eth8: CTRL-EVENT-EAP-STARTED EAP authentication started eth8: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=13 eth8: CTRL-EVENT-EAP-METHOD EAP vendor 0 method 13 (TLS) selected eth8: CTRL-EVENT-EAP-PEER-CERT depth=2 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Root CA' hash=REDACTED eth8: CTRL-EVENT-EAP-PEER-CERT depth=1 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Enhanced Services CA' hash=REDACTED eth8: CTRL-EVENT-EAP-PEER-CERT depth=0 subject='/C=US/ST=Michigan/L=Southfield/O=ATT Services Inc/OU=OCATS/CN=aut00asm.asm.bellsouth.net' hash=REDACTED eth8: CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:aut00asm.asm.bellsouth.net

This is a fresh setup, I just got the UDM-Pro today. Coming from a USG-Pro-4 using the eap_proxy bypass, I was really hoping to have this working. I got the certs from an eBay seller, does anyone know if AT&T have things locked down by MAC? Would I need to call them to get them to whitelist the new MAC address?

Answer 45 · 2020-11-05T02:33:50.000Z

@wjhanna I pulled my certs directly off my AT&T router, so I don't think that's the problem.

Answer 46 · 2021-05-20T01:39:53.000Z

I've got a problem similar to @wjhanna, where I don't get an IP address and the logs show the authentication looping.

Some additional info: docker restart wpa_supplicant-udmpro && docker logs -f wpa_supplicant-udmpro spits out

eth8: CTRL-EVENT-EAP-STARTED EAP authentication started
eth8: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=13
eth8: CTRL-EVENT-EAP-METHOD EAP vendor 0 method 13 (TLS) selected
eth8: CTRL-EVENT-EAP-PEER-CERT depth=2 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Root CA' hash=REDACTED
eth8: CTRL-EVENT-EAP-PEER-CERT depth=1 subject='/C=US/O=ATT Services Inc/CN=ATT Services Inc Enhanced Services CA' hash=REDACTED
eth8: CTRL-EVENT-EAP-PEER-CERT depth=0 subject='REDACTED' hash=REDACTED
eth8: CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:REDACTED

very quickly, but

eth8: CTRL-EVENT-EAP-SUCCESS EAP authentication completed successfully
eth8: CTRL-EVENT-CONNECTED - Connection to 01:80:c2:00:00:03 completed [id=0 id_str=]

takes forever to be printed, and eth8: CTRL-EVENT-EAP-STARTED EAP authentication started through eth8: CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:REDACTED is immediately printed again.

Things I've tried:

Setting the MAC address of eth8 to the MAC in wpa_supplicant.conf. No apparent change.
Setting eapol_version in wpa_supplicant.conf to 2 instead of 1. No apparent change.
Setting identity to the ONT-facing MAC of my Pace 5268ac. No apparent change.
Setting identity to the MAC of the ONT. No apparent change.

Potentially important info:

My certificate files came from an NVG510, but I have FTTH.
I've never tried the wpa_supplicant method of bypassing the RG before - before getting my UDM Pro, I used eap_proxy on an ER4.
I have no idea if XGS-PON has been deployed to my area, but I live in a relatively new neighborhood.
At the moment, I'm successfully using pbrah/eap_proxy-udmpro to bypass the NAT table of my RG.

Answer 47 · 2022-12-30T14:20:13.000Z

So after all this thread and two years later I decided to try this again. I went back into my old config and noticed my last try had my -d flag after the image name, thus not running the docker in detached mode. Everything is running perfectly for me now and through reboots and all. Thanks everyone for the help.