tcplanz: A Python repository from MichaelMonashev

What is this for

This project allows to get web server performance metric from traffic TCPDump. This allows to measure up metric like html delivery time which are usually not available in runtime and have second opinion about other metrics like server latency.

Installation

This is not meant to be used directy on frontend, because it could consume lot of memory and CPU parsing pcap files. Please install on development server or workstation

Before installation you need python (pypy is preferrable because it faster on task like this). Only other dependance is dpkt (python tcpdump parsing library) see https://dpkt.readthedocs.org If you use pip to install dpkt please say pypy -m pip install dpkt instead of pip install dpkt, second spell installs dpkt for python not pypy.

How to get data

You should dump on frontend/reverse proxy/balancer i.e. something which holds tcp connections directly to users

Simple way: sudo tcpdump -w data.pcap port 80

Production way:

use nice -10 to give additional priority to tcpdump
use -B option for largest buffer available on your system. Sometimes tcpdump drop some packets because disk is busy
use -C option to split file by files of reasonable size (1-10Gb) this is easier for copying/etc
use -z to compress splitted files after writing, note: nice/ionice gzip to prevent it from preemping tcpdump
dump only traffic you need, i.e. use filters "port 80 or port 8080"
if you debugging frontend itself consider dumping both traffic to user and backend. You may find something interesting comparing timings.

Parsing

   parse-pcap.py <outdir> (parse|split|sparse) <input files shoild have extensions .pcap or .pcap.gz> 

   outdir - directory where http.txt and debug.txt will be created.

   parse  - just parse files. will use lot of memory because every active session is kept in memory until closed. 
            and tcpdump has a lot of sessions which never ends. Session which crossed tcpdump files will be preserved.
            order of files is important, i.e. better to have timestamp or number inside file name.
   split  - extract tcp sessions from pcap files split it to several new files. Each session will be only in one file.
            order of packets will be changed. I.e. session 1 - packets 1..N, session 2 packets N+1..M, etc.
            timestamps will be preserved. 
   sparse - parse splited files. Actually it is same as just run programs on all files one by one.
            sessions across files will be separated.

Examples:

     parse-pcap.py out parse tcpdump.pcap.gz 
     (test example)

     parse-pcap.py splitted split 100Gb-dump/*.pcap.gz
     parse-pcap.py out sparse splitted/*.pcap.gz 

     (real world example. If you want to parse 100Gb of tcpdump you may need comparable amount of memory for split 
     operation. Comparable means 100Gb if all http sessions is keep-alive and not finished inside file. And small amount of memory if all sessions are short. 
     In real world you have mixed traffic so be ready to have 20Gb-30Gb of memory for average large service.
     If you need only some ports save memory by please patching string in decode-pcap.py 
        "ports = None #set([80,8080]) #uncomment this if yuo don't need all traffic "

     In yandex we have used map/reduce version of this program, it were using proprietary map/ruduce implementation, 
     so it useless outside. But I waht to port in to Hadoop, if you have lot of data and test stand ask me)

After parsing there will be files http.txt and debug.txt

Output format

Columns of http.txt are the folowing:

server  
server_port     
client  
client_port     
request                 - GET/POST/HEAD
uri 
response                - HTTP
status                  - 200/301/500/etc
reqid                   - Yandex Specific request id. Will be None in your case. 

request_start_time      - timestamp, first packet of request
request_end_time        - timestamp, last packet of request, usually same as first, unless it is large POST request.

response_start_time     - timestamp, first packet of response
response_end_time       - timestamp, last packet of response   

request_start_acked     - time then your server acked request first packet
request_end_acked       - time then your server acked request last packet      

response_start_acked    - time then user acked delivery of start of data
response_end_acked      - time then user acked delivery of end of data

request_retr            - number of retransmission in request     
request_false_retr      - broken   
request_keepalive_retr  - always 0, we track keepalive retransmission but they are not a part of request 


response_retr           - number of retransmission in response
response_false_retr     - broken
response_keepalive_retr - always 0, we track keepalive retransmission but they are not a part of response



request_min_rtt         - different round trip time statistic by seq/ack
request_median_rtt      
request_max_rtt 
response_min_rtt        
response_median_rtt     
response_max_rtt        
req_avg_retr_time       
resp_avg_retr_time      


reqpackets              - number of packets in request. Usually 1.
reqlen                  - length of payload of request
reqreallen              - length of payload without retransmission.


resppackets             - number of packets in response.
resplen                 - length of payload of response
respreallen             - length of payload without retransmission.


req_user_agent          - user agent. Split by this to distinguish mobile/nonmobile traffic
req_host                - host from HTTP protocol


server_as               - broken
server_24mask           - broken
client_as               - broken 
client_24mask           - broken

Intersing thing to calculate yourself