MarcJHuber/event-driven-servers

Tacacs behind AWS NLB

Closed this issue · 6 comments

djerihovs commented

Hello,

I have problem running Tacacs-ng behind AWS NLB load-balancer with haproxy = yes in spawned config. Traffic is reaching Tacacs server but in debug log is see following message:
883: 10:05:30.784 0/00000000: - 10.64.208.56: Illegal major version specified: found 13 wanted 192

YFYI: If i try to put Tacacs behind Haproxy it works without problems.

Config:

#!/usr/local/sbin/tac_plus-ng
id = spawnd {
    listen = {
        port = 4948
        haproxy = yes
    }
}

AWS NLB config:
Proxy protocol v2 - enabled

MarcJHuber commented

Hi,

"13" is actually the first octet of the proxy v2 protocol signature. Could you run a packet capture and check whether the proxy v2 protocol header matches the v2 format, and especially whether the complete header is part of the initial TCP segment?

Thanks,

Marc

djerihovs commented

Based on what i see from tcpdump it does have v2 format in headers (0d0a 0d0a 000d 0a51 5549 540a) i can even decode and see what it has source IP of network device. Strange thing what in some cases when i start tac_plus-ng process in debug mode i don't see error about Illegal major version specified and actually nothing in debug output.

End of debug output look like this (process started like this: tac_plus-ng /etc/tac_plus-ng.cfg -f -d -1):

321: file=/etc/tac_plus-ng.cfg line=221 sym=[}] buf='}'
321: file=/etc/tac_plus-ng.cfg line=224 sym=[<end-of-file>] buf=''
321: 21:46:15.192 0/00000000: - Version 9acb4e4c514cf2f55c339d46b2bcd36ffacab4b4 initialized


[root@tools-test-tacacs-6788c9c6f6-l8gp9 /]# tcpdump -i any -nnvvXSs 1514 port 49                  
tcpdump: data link type **LINUX_SLL2**
dropped privs to tcpdump
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 1514 bytes
21:33:53.366185 eth0  In  IP (tos 0x0, ttl 59, id 7442, offset 0, flags [DF], proto TCP (6), length 64)
    10.64.208.56.41469 > 10.130.7.41.49: Flags [S], cksum 0xf1ac (correct), seq 2030627057, win 65535, options [mss 1460,nop,wscale 1,nop,nop,sackOK,nop,nop,TS val 386481406 ecr 0], length 0
        0x0000:  4500 0040 1d12 4000 3b06 3683 0a40 d038  E..@..@.;.6..@.8
        0x0010:  0a82 0729 a1fd 0031 7908 e8f1 0000 0000  ...)...1y.......
        0x0020:  b002 ffff f1ac 0000 0204 05b4 0103 0301  ................
        0x0030:  0101 0402 0101 080a 1709 3cfe 0000 0000  ..........<.....
21:33:53.366197 eth0  Out IP (tos 0x0, ttl 127, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.130.7.41.49 > 10.64.208.56.41469: Flags [S.], cksum 0xec51 (incorrect -> 0x678e), seq 2915407953, ack 2030627058, win 62643, options [mss 8961,sackOK,TS val 3555749892 ecr 386481406,nop,wscale 7], length 0
        0x0000:  4500 003c 0000 4000 7f06 0f99 0a82 0729  E..<..@........)
        0x0010:  0a40 d038 0031 a1fd adc5 9851 7908 e8f2  .@.8.1.....Qy...
        0x0020:  a012 f4b3 ec51 0000 0204 2301 0402 080a  .....Q....#.....
        0x0030:  d3f0 7004 1709 3cfe 0103 0307            ..p...<.....
21:33:53.393802 eth0  In  IP (tos 0x0, ttl 59, id 7443, offset 0, flags [DF], proto TCP (6), length 52)
    10.64.208.56.41469 > 10.130.7.41.49: Flags [.], cksum 0x2625 (correct), seq 2030627058, ack 2915407954, win 33304, options [nop,nop,TS val 386481436 ecr 3555749892], length 0
        0x0000:  4500 0034 1d13 4000 3b06 368e 0a40 d038  E..4..@.;.6..@.8
        0x0010:  0a82 0729 a1fd 0031 7908 e8f2 adc5 9852  ...)...1y......R
        0x0020:  8010 8218 2625 0000 0101 080a 1709 3d1c  ....&%........=.
        0x0030:  d3f0 7004                                ..p.
21:33:53.434700 eth0  In  IP (tos 0x0, ttl 59, id 7445, offset 0, flags [DF], proto TCP (6), length 152)
    10.64.208.56.41469 > 10.130.7.41.49: Flags [P.], cksum 0xff8c (correct), seq 2030627058:2030627158, ack 2915407954, win 33304, options [nop,nop,TS val 386481476 ecr 3555749892], length 100
        0x0000:  4500 0098 1d15 4000 3b06 3628 0a40 d038  E.....@.;.6(.@.8
        0x0010:  0a82 0729 a1fd 0031 7908 e8f2 adc5 9852  ...)...1y......R
        0x0020:  8018 8218 ff8c 0000 0101 080a 1709 3d44  ..............=D
        0x0030:  d3f0 7004 0d0a 0d0a 000d 0a51 5549 540a  ..p........QUIT.
        0x0040:  2111 0054 0a0a 0014 0a40 d038 c689 0031  !..T.....@.8...1
        0x0050:  0300 04a6 b09e d304 003e 0000 0000 0000  .........>......
        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0090:  0000 0000 0000 0000                      ........
21:33:53.434719 eth0  Out IP (tos 0x0, ttl 127, id 50837, offset 0, flags [DF], proto TCP (6), length 52)
    10.130.7.41.49 > 10.64.208.56.41469: Flags [.], cksum 0xec49 (incorrect -> 0xa584), seq 2915407954, ack 2030627158, win 489, options [nop,nop,TS val 3555749960 ecr 386481476], length 0
        0x0000:  4500 0034 c695 4000 7f06 490b 0a82 0729  E..4..@...I....)
        0x0010:  0a40 d038 0031 a1fd adc5 9852 7908 e956  .@.8.1.....Ry..V
        0x0020:  8010 01e9 ec49 0000 0101 080a d3f0 7048  .....I........pH
        0x0030:  1709 3d44                                ..=D
21:33:53.434701 eth0  In  IP (tos 0x0, ttl 59, id 7445, offset 0, flags [DF], proto TCP (6), length 83)
    10.64.208.56.41469 > 10.130.7.41.49: Flags [P.], cksum 0x6ea8 (correct), seq 2030627158:2030627189, ack 2915407954, win 33304, options [nop,nop,TS val 386481476 ecr 3555749892], length 31
        0x0000:  4500 0053 1d15 4000 3b06 366d 0a40 d038  E..S..@.;.6m.@.8
        0x0010:  0a82 0729 a1fd 0031 7908 e956 adc5 9852  ...)...1y..V...R
        0x0020:  8018 8218 6ea8 0000 0101 080a 1709 3d44  ....n.........=D
        0x0030:  d3f0 7004 c001 0100 77f9 ea94 0000 0013  ..p.....w.......
        0x0040:  edea 5696 33d5 153b c983 ed65 7164 92f8  ..V.3..;...eqd..
        0x0050:  404e 0a                                  @N.
21:33:53.434722 eth0  Out IP (tos 0x0, ttl 127, id 50838, offset 0, flags [DF], proto TCP (6), length 52)
    10.130.7.41.49 > 10.64.208.56.41469: Flags [.], cksum 0xec49 (incorrect -> 0xa565), seq 2915407954, ack 2030627189, win 489, options [nop,nop,TS val 3555749960 ecr 386481476], length 0
        0x0000:  4500 0034 c696 4000 7f06 490a 0a82 0729  E..4..@...I....)
        0x0010:  0a40 d038 0031 a1fd adc5 9852 7908 e975  .@.8.1.....Ry..u
        0x0020:  8010 01e9 ec49 0000 0101 080a d3f0 7048  .....I........pH
        0x0030:  1709 3d44                                ..=D
21:33:53.434749 eth0  Out IP (tos 0x0, ttl 127, id 50839, offset 0, flags [DF], proto TCP (6), length 52)
    10.130.7.41.49 > 10.64.208.56.41469: Flags [R.], cksum 0xec49 (incorrect -> 0xa561), seq 2915407954, ack 2030627189, win 489, options [nop,nop,TS val 3555749960 ecr 386481476], length 0
        0x0000:  4500 0034 c697 4000 7f06 4909 0a82 0729  E..4..@...I....)
        0x0010:  0a40 d038 0031 a1fd adc5 9852 7908 e975  .@.8.1.....Ry..u
        0x0020:  8014 01e9 ec49 0000 0101 080a d3f0 7048  .....I........pH
        0x0030:  1709 3d44
djerihovs commented

Sorry about misdirecting you error about Illegal major version specified is shown if i enable haproxy auto-detect = yes on realm level if i enable haproxy = yes on spawnd level i don't receive anything in debug log.

Debug output with haproxy auto-detect = yes:

146: 23:09:52.738 0/00000000: - Version 9acb4e4c514cf2f55c339d46b2bcd36ffacab4b4 initialized
145: 23:09:58.070 0/00000000: - connection request from 10.64.208.56 (realm: haproxy)
145: 23:09:58.070 0/00000000: - 10.64.208.56: Illegal major version specified: found 13 wanted 192


#!/usr/local/sbin/tac_plus-ng
id = spawnd {
    listen = {
        port = 49
        realm = haproxy
    }
}

debug redirect = /dev/stdout
id = tac_plus-ng {
    realm haproxy {
        haproxy auto-detect = yes
    }
MarcJHuber commented

Hi,

thanks for the packet dump, I'll push a fix in a couple of minutes. AWS NLB adds some extra padding which is legal but unexpected.
Cheers,

Marc

  • 👍1
  • 🎉1
djerihovs commented

Thank you very much issue is resolved, i checked now everything is working as expected

MarcJHuber commented

Hi,

great, thanks a lot for reporting and helping with this issue!

Cheers,

Marc