Invalid characters in the server name (http/TLS).
vel21ripn opened this issue · 5 comments
There are two situations that IMHO need to be corrected.
The http protocol parser copies the data from the "Host:" header without checking.
Traffic analysis showed that some clients add a space character after the server name.
The second situation is more interesting.
In the tls protocol, the "SNI" field contains text with line feed characters.
tcpdump
0x00a0: 4f00 004c 206c 696b 6520 4765 636b 6f29 O..L.like.Gecko)
0x00b0: 2043 6872 6f6d 652f 3132 302e 302e 302e .Chrome/120.0.0.
0x00c0: 3020 5361 6661 7269 2f35 3337 2e33 360a 0.Safari/537.36.
0x00d0: 4163 6365 7074 3a20 2f0a 436f 6e6e 6563 Accept:./.Connec
0x00e0: 7469 6f6e 3a20 6b65 6570 2d61 6c69 7665 tion:.keep-alive
0x00f0: 0017 0000 0023 0092 1179 618e a6fe d21c .....#...ya.....
ndpiReader with DEBUG_TLS
Reading packets from pcap file p440-s.pcap...
Running thread 0...
==>> ndpi_search_tls_wrapper() 0 [len: 413][version: 0]
[TLS] Processing block 1
TLS _processClientServerHello() called
TLS [len: 408][handshake_type: 01]
Client TLS [client cipher_len: 28][tls_version: 0x0303]
Client TLS [non-GREASE cipher suite: 49195/0xC02B] [0/28]
Client TLS [cipher suite: 49195/0xC02B] [0/28]
Client TLS [non-GREASE cipher suite: 49199/0xC02F] [2/28]
Client TLS [cipher suite: 49199/0xC02F] [2/28]
Client TLS [non-GREASE cipher suite: 52393/0xCCA9] [4/28]
Client TLS [cipher suite: 52393/0xCCA9] [4/28]
Client TLS [non-GREASE cipher suite: 52392/0xCCA8] [6/28]
Client TLS [cipher suite: 52392/0xCCA8] [6/28]
Client TLS [non-GREASE cipher suite: 52244/0xCC14] [8/28]
Client TLS [cipher suite: 52244/0xCC14] [8/28]
Client TLS [non-GREASE cipher suite: 52243/0xCC13] [10/28]
Client TLS [cipher suite: 52243/0xCC13] [10/28]
Client TLS [non-GREASE cipher suite: 49162/0xC00A] [12/28]
Client TLS [cipher suite: 49162/0xC00A] [12/28]
Client TLS [non-GREASE cipher suite: 49172/0xC014] [14/28]
Client TLS [cipher suite: 49172/0xC014] [14/28]
Client TLS [non-GREASE cipher suite: 49161/0xC009] [16/28]
Client TLS [cipher suite: 49161/0xC009] [16/28]
Client TLS [non-GREASE cipher suite: 49171/0xC013] [18/28]
Client TLS [cipher suite: 49171/0xC013] [18/28]
Client TLS [non-GREASE cipher suite: 156/0x009C] [20/28]
Client TLS [cipher suite: 156/0x009C] [20/28]
Client TLS [non-GREASE cipher suite: 53/0x0035] [22/28]
Client TLS [cipher suite: 53/0x0035] [22/28]
Client TLS [non-GREASE cipher suite: 47/0x002F] [24/28]
Client TLS [cipher suite: 47/0x002F] [24/28]
Client TLS [non-GREASE cipher suite: 10/0x000A] [26/28]
Client TLS [cipher suite: 10/0x000A] [26/28]
Client TLS [compression_len: 1]
Client TLS [extensions_len: 303]
Client TLS [extension_id: 65281][extension_len: 1]
Client TLS [extension_offset/len: 5/1]
Client TLS [extension_id: 0][extension_len: 81]
[TLS] Extensions: found server name
[TLS] SNI: [ like gecko) chrome/120.0.0.0 safari/537.36
accept: /
connection: keep-alive]
[TLS] SNI: (NO DGA) [ like gecko) chrome/120.0.0.0 safari/537.36
accept: /
connection: keep-alive]
Client TLS [extension_offset/len: 90/81]
Client TLS [extension_id: 23][extension_len: 0]
Client TLS [extension_offset/len: 94/0]
Client TLS [extension_id: 35][extension_len: 146]
Client TLS [extension_offset/len: 244/146]
Client TLS [extension_id: 13][extension_len: 22]
Client TLS [SIGNATURE_ALGORITHMS: block_len=22/len=20]
Client TLS [SIGNATURE_ALGORITHMS: 060106030501050304010403030103030201020]
Client TLS [extension_offset/len: 270/22]
Client TLS [extension_id: 5][extension_len: 5]
Client TLS [extension_offset/len: 279/5]
Client TLS [extension_id: 18][extension_len: 0]
Client TLS [extension_offset/len: 283/0]
Client TLS [extension_id: 30032][extension_len: 0]
Client TLS [extension_offset/len: 287/0]
Client TLS [extension_id: 11][extension_len: 2]
Client TLS [EllipticCurveFormat: len=2]
Client TLS [EllipticCurveFormat: 0]
Client TLS [extension_offset/len: 293/2]
Client TLS [extension_id: 10][extension_len: 6]
Client TLS [EllipticCurveGroups: len=6]
Client TLS [EllipticCurve: 23/0x0017]
Client TLS [EllipticCurve: 24/0x0018]
Client TLS [extension_offset/len: 303/6]
[TLS] ndpi_int_tls_add_connection()
*** TLS [version: 303][Client Hello]
nDPI Memory statistics:
nDPI Memory (once): 37.51 KB
Flow Memory (per flow): 1000 B
Actual Memory: 5.08 MB
Peak Memory: 5.08 MB
Setup Time: 36 msec
Packet Processing Time: 0 msec
Traffic statistics:
Ethernet bytes: 18109 (includes ethernet CRC/IFC/trailer)
Discarded bytes: 0
IP packets: 20 of 20 packets total
IP bytes: 17629 (avg pkt size 881 bytes)
Unique flows: 1
TCP Packets: 20
UDP Packets: 0
VLAN Packets: 0
MPLS Packets: 0
PPPoE Packets: 0
Fragmented Packets: 0
Max Packet size: 2940
Packet Len < 64: 9
Packet Len 64-128: 1
Packet Len 128-256: 1
Packet Len 256-1024: 3
Packet Len 1024-1500: 1
Packet Len > 1500: 5
nDPI throughput: 40.98 K pps / 283.12 Mb/sec
Analysis begin: 30/Jan/2024 14:16:15
Analysis end: 30/Jan/2024 14:16:16
Traffic throughput: 34.89 pps / 246.79 Kb/sec
Traffic duration: 0.573 sec
DPI Packets (TCP): 5 (5.00 pkts/flow)
Confidence: DPI 1 (flows)
Detected protocols:
TLS packets: 20 bytes: 17629 flows: 1
Protocol statistics:
Safe packets: 20 bytes: 17629 flows: 1
Risk stats [found 1 (100.0 %) flows with risks]:
Known Proto on Non Std Port 1 [25.0 %]
TLS (probably) Not Carrying HTTPS 1 [25.0 %]
Non-Printable/Invalid Chars Detected 1 [25.0 %]
Possible Exploit 1 [25.0 %]
NOTE: as one flow can have multiple risks set, the sum of the
last column can exceed the number of flows with risks.
JA3 Host Stats:
IP Address # JA3C
1 159.65.45.59 1
1 TCP 159.65.45.59:64316 <-> 10.0.0.1:440 [proto: 91/TLS][IP: 0/Unknown][Encrypted][Confidence: DPI][DPI packets: 5][cat: Web/5][9 pkts/2712 bytes <-> 11 pkts/14917 bytes][Goodput ratio: 81/96][0.57 sec][Hostname/SNI: like gecko) chrome/120.0.0.0 safari/537.36
accept: /
connection: keep-alive][bytes ratio: -0.692 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 60/52 191/190 73/72][Pkt Len c2s/s2c min/avg/max/stddev: 60/60 301/1356 1724/2974 518/1289][Risk: ** Known Proto on Non Std Port **** TLS (probably) Not Carrying HTTPS **** Non-Printable/Invalid Chars Detected **** Possible Exploit **][Risk Score: 310][Risk Info: like gecko) chrome/120.0.0.0 safari/537.36
accept: /
connection: keep-alive / No ALPN][TLSv1.2][JA3C: fa030dbcb2e3c7141d3c2803780ee8db][JA4: t12d141000_20a10634286c_cc51b52e28e4][Firefox][Plen Bins: 40,0,5,0,5,0,0,5,0,0,0,0,5,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5,0,28]
I think not everyone is ready to handle a hostname with control characters.
What's the best way to deal with control characters in a hostname? Remove or replace?
It is possible to fix both problems by making changes to the ndpi_hostname_sni_set() function.
I agree that we should fix that; we might do something similar of what we are already doing in dns: first "normalize" the name and then save it as metadata.
@vel21ripn, out of curiosity, could you share the flow triggering this output:
1 TCP 159.65.45.59:64316 <-> 10.0.0.1:440 [proto: 91/TLS][IP: 0/Unknown][Encrypted][Confidence: DPI][DPI packets: 5][cat: Web/5][9 pkts/2712 bytes <-> 11 pkts/14917 bytes][Goodput ratio: 81/96][0.57 sec][Hostname/SNI: like gecko) chrome/120.0.0.0 safari/537.36
accept: /
It seems suspicious that we have that kind of string in the SNI (because it seems an HTTP user agent...)
The question is how to normalize the name: remove invalid characters or replace them. If replaced, then with what?
Spaces at the end of a line can definitely be removed.
This traffic is not unique. DigitalOcean has several servers. They all use TCP port 440. All clients have HTTP headers instead of SNI, but not always the same.
It's very similar to a VPN.
The question is how to normalize the name: remove invalid characters or replace them. If replaced, then with what? Spaces at the end of a line can definitely be removed.
In dns we replace invalid characters :
if(character is not valid) {
if (ndpi_isprint(character) == 0) {
return '?';
} else {
return '_';
}
}
I think that we should keep the same logic
This traffic is not unique. DigitalOcean has several servers. They all use TCP port 440. All clients have HTTP headers instead of SNI, but not always the same. It's very similar to a VPN.
Could you share a pcap?
Excellent logic for replacing invalid characters!
Traffic examples
p440-1.pcap.gz
p440-2.pcap.gz