logstash-plugins/logstash-codec-netflow

Problem with zero-filled export packets

maxcaines opened this issue · 8 comments

maxcaines commented

  • Version: 4.0.1

  • Operating System: Ubuntu 14.04 LTS

  • Config File (if you have sensitive info, please remove it):

  • Sample Data: see attached packet capture
    invalid-netflow.zip

  • Steps to Reproduce: send Netflow (v9) records from Palo Alto 8.1 firewall. All packets seem to contain zero bytes following the flowsets, and all recent versions of the codec reject these packets. Looking at the Netflow9PDU class in netflow/util.rb that extracts the flow records, it expects valid flow records to fill the packet, whereas the Netflow5PDU class expects as many records as are specified by the count in the packet header. The RFC doesn’t mention the possibility of data in the packet after the last flow record, but that’s what Palo Alto generates. It would be useful if the v9 class could ignore it. It generates a warning in the Logstash log about the invalid flowset length (0) and drops the packet.

jorritfolmer commented

Thanks for the pcap, I'll look into it later.

maxcaines commented

If it's any help, I've verified that Logstash 2.3.2 with version 2.1.0 of the codec works fine on records from our Palo Alto firewall. However, I can't use that version with Logstash 5. I'll see if I can work out what the relevant difference is in the code

maxcaines commented

Hi Jorrit

I may have a solution to this. I enclose updated copies of netflow.rb and util.rb. They pass all the "rspec" tests, and I've been running this version of the plugin for several days receiving Netflow data from our Palo Alto without issues.

The approach I've taken is:

  1. The only way to avoiding reading the zero padding as an extra flowset would be to count the individual flow records and stop when the total reached the value in the flow_records field. But for data flowsets that requires knowledge of the lengths of the flow records, and that information is in the templates. I can't come up with any way to get at that information in declarative code, so I have to create an extra flowset with a zero flowset_length. This is checked for and ignored in netflow.rb.
  2. The zero padding looks like an extra flowset with a zero flowset_length field, I've changed the "assert" in util.rb that checks its validity to allow a zero value.
  3. The padding looks like a template flowset, because the flowset_id is zero. I have changed the template record definition in util.rb to stop reading flow records if flowset_length is zero. I've also added an onlyif field which reads the rest of the packet in that case.

The overall effect of these changes should accommodate the padding that Palo Alto devices add to the Netflow packets, but still leave most of the checks on invalid packets that were added on 27/4/16. The code base I started from is 4.0.1

I hope this is of use

Max

netflow.zip

jorritfolmer commented

Thanks for investigating!

So the root cause appears to be L418 of util.rb
This causes the codec to keep reading because the PA packets are all 1442 bytes even though some only contain 1 record with 160 bytes.

Changing :records, :read_until => :eof to :initial_length => :flow_records works to address the root cause, but breaks other stuff. Not sure why yet. It may be this exposed other hacky workarounds.

maxcaines commented
jorritfolmer commented

Thanks, good point!

jorritfolmer commented

v4.0.2 includes your fix.

maxcaines commented