tcpdp

tcpdp is TCP dump tool with custom dumper and structured logger written in Go.

tcpdp has 3 modes:

TCP Proxy server mode
Probe mode ( using libpcap )
Read pcap file mode

Usage

`tcpdp proxy` : TCP proxy server mode

$ tcpdp proxy -l localhost:12345 -r localhost:1234 -d hex # hex.Dump()

$ tcpdp proxy -l localhost:55432 -r db.internal.example.com:5432 -d pg # Dump query of PostgreSQL

$ tcpdp proxy -l localhost:33306 -r db.example.com:3306 -d mysql # Dump query of MySQL

With server-starter

https://github.com/lestrrat-go/server-starter

$ start_server --port 33306 -- tcpdp proxy -s -r db.example.com:3306 -d mysql

With config file

$ tcpdp proxy -c config.toml

`tcpdp probe` : Probe mode (like tcpdump)

$ tcpdp probe -i lo0 -t localhost:3306 -d mysql # is almost the same setting as 'tcpdump -i lo0 host 127.0.0.1 and tcp port 3306'

$ tcpdp probe -i eth0 -t 3306 -d hex # is almost the same setting as 'tcpdump -i eth0 tcp port 3306'

`tcpdp read` : Read pcap file mode

$ tcpdump -i eth0 host 127.0.0.1 and tcp port 3306 -w mysql.pcap
$ tcpdp read mysql.pcap -d mysql -t 3306 -f ltsv

`tcpdp config` Create config

$ tcpdp config > myconfig.toml

Show current config

$ tcpdp config

config format

[tcpdp]
pidfile = "/var/run/tcpdp.pid"
dumper = "mysql"

[probe]
target = "db.example.com:3306"
interface = "en0"
bufferSize = "2MB"
immediateMode = false
snapshotLength = "auto"
internalBufferLength = 10000
filter = ""

[proxy]
useServerStarter = false
listenAddr = "localhost:3306"
remoteAddr = "db.example.com:3306"

[log]
dir = "/var/log/tcpdp"
enable = true
enableInternal = true
stdout = true
format = "ltsv"
rotateEnable = true
rotationTime = "daily"
rotationCount = 7
# You can execute arbitrary commands after rotate
# $1 = prev filename
# $2 = current filename
rotationHook = "/path/to/after_rotate.sh"
fileName = "tcpdp.log"

[dumpLog]
dir = "/var/log/dump"
enable = true
stdout = false
format = "json"
rotateEnable = true
rotationTime = "hourly"
rotationCount = 24
fileName = "dump.log"

Installation

$ go get github.com/k1LoW/tcpdp

Architecture

tcpdp proxy connection diagram

      client_addr
           ^
           |        tcpdp
+----------|---------------+
|          v               |
|  proxy_listen_addr       |
|         + ^              |
|         | |   +--------+ |
|         |<----+ dumper | |
|         | |<--+        | |
|         | |   +--------+ |
|         v +              |
|  proxy_client_addr       |
|          ^               |
+----------|---------------+
           |
           v
      remote_addr

tcpdp probe connection diagram

                    server
+--------------------------+
|                          |
|                      +---+---+
|       <--------------| eth0  |----------->
|            interface +---+---+
|            /target     ^ |
|                        | |
|         tcpdp          | |
|        +--------+      | |
|        | dumper +------+ |
|        +--------+        |
+--------------------------+

tcpdp read diagram

                  tcpdp
+--------+ STDIN +--------+ STDOUT
| *.pcap +------>+ dumper +-------->
+--------+       +--------+

tcpdp.log ( `tcpdp proxy` or `tcpdp probe` )

key	description	mode
ts	timestamp	proxy / probe / read
level	log level	proxy / probe
msg	log message	proxy / probe
error	error info	proxy / probe
caller	error caller	proxy / probe
conn_id	TCP connection ID by tcpdp	proxy / probe
target	probe target	proxy / probe
dumper	dumper type	proxy / probe
use_server_starter	use server_starter	proxy
conn_seq_num	TCP comunication sequence number by tcpdp	proxy
client_addr	client address	tcpdp.log, hex, mysql, pg
remote_addr	remote address	proxy
proxy_listen_addr	listen address	proxy
direction	client to remote: `->` / remote to client: `<-`	proxy
interface	probe target interface	probe
mtu	interface MTU (Maximum Transmission Unit)	probe
mss	TCP connection MSS (Max Segment Size)	probe
probe_target_addr	probe target address	probe
filter	BPF (Berkeley Packet Filter)	probe
buffer_size	libpcap buffer_size	probe
immediate_mode	libpcap immediate_mode	probe
snapshot_length	libpcap snapshot length	probe
internal_buffer_length	tcpdp internal packet buffer length	probe

Dumper

mysql

MySQL query dumper

NOTICE: MySQL query dumper require --target option when tcpdp proxy tcpdp probe

key	description	mode
ts	timestamp	proxy / probe / read
conn_id	TCP connection ID by tcpdp	proxy / probe / read
conn_seq_num	TCP comunication sequence number by tcpdp	proxy
client_addr	client address	proxy
proxy_listen_addr	listen address	proxy
proxy_client_addr	proxy client address	proxy
remote_addr	remote address	proxy
direction	client to remote: `->` / remote to client: `<-`	proxy
interface	probe target interface	probe
src_addr	src address	probe / read
dst_addr	dst address	probe / read
probe_target_addr	probe target address	probe
proxy_protocol_src_addr	proxy protocol src address	probe / proxy /read
proxy_protocol_dst_addr	proxy protocol dst address	probe / proxy /read
query	SQL query	proxy / probe / read
stmt_id	statement id	proxy / probe / read
stmt_prepare_query	prepared statement query	proxy / probe / read
stmt_execute_values	prepared statement execute values	proxy / probe / read
character_set	character set	proxy / probe / read
username	username	proxy / probe / read
database	database	proxy / probe / read
seq_num	sequence number by MySQL	proxy / probe / read
command_id	command_id for MySQL	proxy / probe / read

pg

PostgreSQL query dumper

NOTICE: PostgreSQL query dumper require --target option tcpdp proxy tcpdp probe

key	description	mode
ts	timestamp	proxy / probe / read
conn_id	TCP connection ID by tcpdp	proxy / probe / read
conn_seq_num	TCP comunication sequence number by tcpdp	proxy
client_addr	client address	proxy
proxy_listen_addr	listen address	proxy
proxy_client_addr	proxy client address	proxy
remote_addr	remote address	proxy
direction	client to remote: `->` / remote to client: `<-`	proxy
interface	probe target interface	probe
src_addr	src address	probe / read
dst_addr	dst address	probe / read
probe_target_addr	probe target address	probe
proxy_protocol_src_addr	proxy protocol src address	probe / proxy /read
proxy_protocol_dst_addr	proxy protocol dst address	probe / proxy /read
query	SQL query	proxy / probe / read
portal_name	portal Name	proxy / probe / read
stmt_name	prepared statement name	proxy / probe / read
parse_query	prepared statement query	proxy / probe / read
bind_values	prepared statement bind(execute) values	proxy / probe / read
username	username	proxy / probe / read
database	database	proxy / probe / read
message_type	message type for PostgreSQL	proxy / probe / read

hex

key	description	mode
ts	timestamp	proxy / probe / read
conn_id	TCP connection ID by tcpdp	proxy / probe / read
conn_seq_num	TCP comunication sequence number by tcpdp	proxy
client_addr	client address	proxy
proxy_listen_addr	listen address	proxy
proxy_client_addr	proxy client address	proxy
remote_addr	remote address	proxy
direction	client to remote: `->` / remote to client: `<-`	proxy
interface	probe target interface	probe
src_addr	src address	probe / read
dst_addr	dst address	probe / read
probe_target_addr	probe target address	probe
proxy_protocol_src_addr	proxy protocol src address	probe / proxy /read
proxy_protocol_dst_addr	proxy protocol dst address	probe / proxy /read
bytes	bytes string by hex.Dump	proxy / probe / read
ascii	ascii string by hex.Dump	proxy / probe / read

k1LoW/tcpdp