timpalpant/go-iex

pcap2json Unknown magic a0d0d0a

MoorsTech opened this issue · 1 comments

MoorsTech commented

Hi there!

I really just wanted to checkout whats contained in the pcap files returned from the https://api.iextrading.com/1.0/hist endpoint.
I was testing the pcap2json utility with the example file available here:
https://www.googleapis.com/download/storage/v1/b/iex/o/data%2Ffeeds%2F20170911%2F20170911_IEXTP1_TOPS1.5.pcap.gz?generation=1505174874390598&alt=media

Output as follows:

pcap2json < 20170911_IEXTP1_TOPS1.5.pcap
2017/09/12 15:01:03 Unknown magic a0d0d0a

Had comparable errors with other pcap parsers. Wireshark seems to parse it just fine but I've got no clue how to decode the raw payload <.<
I'm on ubuntu 16.04 LTS using go 1.9.

timpalpant commented

Hmm, newer files seem to have some kind of merge header that the older ones (for instance https://www.googleapis.com/download/storage/v1/b/iex/o/data%2Ffeeds%2F20170103%2F20170103_IEXTP1_TOPS1.5.pcap.gz?generation=1495053678124095&alt=media) didn't have:

$ head -n 25 20170911_IEXTP1_TOPS1.5.pcap


M<+?????????File created by merging: 
File1: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_151415_TOPS1.5A.pcap 
File2: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_144415_TOPS1.5A.pcap 
File3: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_164416_TOPS1.5A.pcap 
File4: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_171416_TOPS1.5A.pcap 
File5: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_124415_TOPS1.5A.pcap 
File6: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_074415_TOPS1.5A.pcap 
File7: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_131415_TOPS1.5A.pcap 
File8: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_154415_TOPS1.5A.pcap 
File9: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_141415_TOPS1.5A.pcap 
File10: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_071415_TOPS1.5A.pcap 
File11: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_121415_TOPS1.5A.pcap 
File12: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_134415_TOPS1.5A.pcap 
File13: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_091415_TOPS1.5A.pcap 
File14: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_084415_TOPS1.5A.pcap 
File15: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_111415_TOPS1.5A.pcap 
File16: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_104415_TOPS1.5A.pcap 
File17: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_094415_TOPS1.5A.pcap 
File18: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_161415_TOPS1.5A.pcap 
File19: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_081415_TOPS1.5A.pcap 
File20: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_114415_TOPS1.5A.pcap 
File21: /var/sre/mnt/nexenta/trading/603_pcaps/2017/20170911_101415_TOPS1.5A.pcap 
mergecap\??6Unknown/not available in original file format(libpcap)	\t?XX?;(R

It looks like the pcap dumps are formatted as pcap-ng (newer Wireshark format) instead of pcap now.

Fortunately, someone has already written a pcapng package, so I've added support for the new filetype in e9356d7.

  • 👍1