TLS not detected
vel21ripn opened this issue · 15 comments
An issue with TLS detection has been found due to extra null bytes in the packet.
This is an "ACK only" packet that cannot contain data.
1 16:14:50.730262 IP (tos 0x0, ttl 125, id 20715, offset 0, flags [DF], proto TCP (6), length 52)
111.111.111.111.58155 > 52.18.127.189.443: Flags [S], cksum 0x7576 (correct), seq 8029850, win 64240,
options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
2 16:14:50.780521 IP (tos 0x0, ttl 233, id 0, offset 0, flags [DF], proto TCP (6), length 52)
52.18.127.189.443 > 111.111.111.111.58155: Flags [S.], cksum 0x9623 (correct), seq 2341660060, ack 8029851, win 26883,
options [mss 1460,nop,nop,sackOK,nop,wscale 8], length 0
3 16:14:50.781367 IP (tos 0x0, ttl 125, id 20716, offset 0, flags [DF], proto TCP (6), length 42)
111.111.111.111.58155 > 52.18.127.189.443: Flags [.], cksum 0x3df7 (correct), seq 8029851:8029853, ack 2341660061, win 513, length 2
4 16:14:50.781920 IP (tos 0x0, ttl 125, id 20717, offset 0, flags [DF], proto TCP (6), length 557)
111.111.111.111.58155 > 52.18.127.189.443: Flags [P.], cksum 0x85ff (correct), seq 8029851:8030368, ack 2341660061, win 513, length 517
5 16:14:50.832249 IP (tos 0x0, ttl 233, id 23901, offset 0, flags [DF], proto TCP (6), length 46)
52.18.127.189.443 > 111.111.111.111.58155: Flags [.], cksum 0x3d81 (correct), seq 2341660061:2341660067, ack 8030368, win 110, length 6
6 16:14:50.832377 IP (tos 0x0, ttl 233, id 23902, offset 0, flags [DF], proto TCP (6), length 153)
52.18.127.189.443 > 111.111.111.111.58155: Flags [P.], cksum 0x4f99 (correct), seq 2341660061:2341660174, ack 8030368, win 110, length 113
The traffic dump is available on google.drive https://drive.google.com/file/d/1oGhkGCqo__M2dTtGly8XW5-4vI6G2HOg/view?usp=sharing
The packets 3 and 4 is overlapped by seq_number!
The packets 5 and 6 is overlapped by seq_number!
If these empty bytes are ignored, then NDPI considers that the fourth packet is "tcp retransmission" :( and the 517 byte packet does not get into the TLS handler.
If these extra bytes are ignored, then TLS is successfully detected.
1 TCP 111.111.111.111:58155 <-> 52.18.127.189:443 [proto: 91/TLS][IP: 265/AmazonAWS][Encrypted][Confidence: DPI][DPI packets: 13][cat: Web/5][28 pkts/6789 bytes <-> 35 pkts/8995 bytes][Goodput ratio: 78/79][130.64 sec][Hostname/SNI: bitrix.info][(Advertised) ALPNs: h2;http/1.1][(Negotiated) ALPN: h2][TLS Supported Versions: GREASE;TLSv1.3;TLSv1.2][bytes ratio: -0.140 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 0/0 5498/4834 45102/45058 12717/11564][Pkt Len c2s/s2c min/avg/max/stddev: 56/60 242/257 1547/2974 352/535][TLSv1.2][JA3C: ab205d804ecf934209c2a1bb94f817e0][JA3S: bfc90d56141386ee83b56cda231cccfc][Chrome][Cipher: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256][Plen Bins: 46,22,6,6,0,0,0,0,0,0,1,0,6,0,1,1,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,1,1,1]
I can't offer a PR yet.
@IvanNardi
You were able to download the file?
I gave you access to the file.
@IvanNardi You were able to download the file? I gave you access to the file.
Yes, I downloaded it.
Could you make it public for everyone, or ,better, attach it here (after having compress it in a zip file)?
@IvanNardi I maked this file public for everyone.
I see that there is quite a lot of such traffic. For 5 minutes about 2300 connections.
This kind of traffic doesn't look random to me, but some kind of evasion technique...
Anyway, there are two different issues here:
- the second packets packet of the "pairs" (4 and 6) are marked as retransmission, but they are not. That 's not a surprise since the generic code in nDPI handling TCP retransmissions and similar issues is quite basic
- even without 1) there are the issue of how to handle overlapping-but-different bytes and that is a policy question, since there is not a right way: in this specific case is quite obvious to an human that we must ignore the first packet and keep the second one, but we can probably create an example the other way around.
snorthas a dedicated configuration parameter to decide exactly how to handle overlapping-but-different bytes and it handle 13 different cases!
Evading firewall/dpi engines/censorship systems with overlapping segments has been a well know technique for decades... unfortunately nDPI lacks any generic logic to handle it.
We could start implementing it (an hard and long task) or look for a quick hack for this specific case: at very least, any additional logic should be configurable and should be disable by default.
Tomorrow I'll take a proper look at you patch
It is highly unlikely that this is an evasion technique.
We see that such packets are transmitted in both directions.
I found 2 links to possible reasons for adding short packets.
https://bugs.centos.org/view.php?id=14435
https://ask.wireshark.org/question/11194/tcp-ack-shows-only-54-bytes-in-wireshark/
Did you find the same pattern in traffic other than TLS? If it is a "generic" issue we should find it with all kinds of TCP traffic (plain HTTP or SSH, for example)
@vel21ripn , could you share some other flows, please? (even TLS)
I will collect a sufficient set of such traffic.
I've only collected "tcp and port 443" traffic so far.
While I'm trying to figure out the reason for the lack of details (ja3s,ja3c) for a large number of connections defined as tls.
I found a program for splitting a .pcap file by tcp connections (pkt2flow), which greatly simplifies the analysis process.
I will collect a sufficient set of such traffic.
Perfect. In the meantime, could you take a look at #1948? It is still a draft, but it should work
I wouldn't rush this PR without a sufficient set of traffic and traffic samples to test.
Statistics showed that padding does not depend on the application protocol.
The most common is the padding of 2 and 6 zero bytes.
The complexity of parsing a sequence in a tcp stream is increased due to SACK
Ok, are you able to collect and share some of these flows?
I am now looking for different samples. This is very slow as it requires a lot of manual work.
There are very difficult partial retransmissions.
I posted typical traffic examples and a utility for analyzing tcp-sequence in my repository in the "pagging" branch.
Look in the utils/tcp_check_seq directory.