rusticata/pcap-parser

Streaming parser example: `error while reading: NomError(Eof)`

kokostek opened this issue · 8 comments

kokostek commented

I was trying to use streaming parser example to count number of blocks in a very large .pcapng file and got this error:

error while reading: NomError(Eof)

I suppose this error should be treated like PcapError::Incomplete case. Not sure though.

Suggesting this change to the example code:

use pcap_parser::*;
use pcap_parser::traits::PcapReaderIterator;
use std::fs::File;
use nom::error::ErrorKind;

let file = File::open(path).unwrap();
let mut num_blocks = 0;
let mut reader = PcapNGReader::new(65536, file).expect("PcapNGReader");
loop {
    match reader.next() {
        Ok((offset, _block)) => {
            println!("got new block");
            num_blocks += 1;
            reader.consume(offset);
        },
        Err(PcapError::Eof) => break,
        Err(PcapError::Incomplete) | Err(PcapError::NomError(ErrorKind::Eof)) => {
            reader.refill().unwrap();
        },
        Err(e) => panic!("error while reading: {:?}", e),
    }
}
println!("num_blocks: {}", num_blocks);
chifflier commented

Hi,
Sorry for the delay in the response.
Does this mean that Eof was returned but the file was not entirely parsed, and you had to ignore the error?

kokostek commented

Hi. Yes, Eof was returned when parser reached the end of circular buffer. But rather than ignore this, I had to refill(). Similar to the Incomplete case.

chifflier commented

That would be a bug, but it will be hard to investigate without more information.
Do you have a way to instrument pcap-parser (maybe print the buffer state / length if the specific condition is reached), or (privately) share some pcap?

kokostek commented

Yes, I can send you the pcap. But I think you can reproduce this on any pcap larger than 65536 bytes. Or maybe reduce size of buffer and try even smaller pcap. Either way, post your contact info and I will share mine some time later.

chifflier commented

This is not related only the file size (I used the crate on files > 150GB). I suspect a specific condition related to the buffer size, and structs alignment (for ex, a header ends exactly at the end of the buffer and there is no content, or something like that)

inferiorhumanorgans commented

BTW I'm seeing the same thing with a 650MB compressed pcapng file. In this case I'm creating a bufreader with the flate2 crate over the file and passing that to the pcap-parser crate. I can duplicate this with smaller files as well. For instance:

https://wiki.wireshark.org/SampleCaptures#Apache_Cassandra

chifflier commented

Thank you for the link, I was able to reproduce the problem.
I'm investigating.

chifflier commented

Found the issue, this was caused by using nom complete parsers instead of streaming.
I'm releasing a new patch release (0.11.1)

  • 👍1