logstash-plugins/logstash-codec-netflow

Can't decode flowset - IPFIX from IXIA packet broker

janniten opened this issue · 14 comments

janniten commented

For all general issues, please provide the following details for fast resolution:

  • Version: 6.3.2
  • Operating System: Redhat
  • Config File (if you have sensitive info, please remove it):
    input {
    udp {
    port => "4001"
    codec => netflow {
    versions => 10
    cache_save_path => "/tmp/netflow_template"
    include_flowset_id => true
    cache_ttl => 999999999
    }
    type => "ipfix"
    }
    }

Hi, I'm trying to ingest netflow data from IXIA packet broker into elastic, but it seems that the plugin cannot find the template in the traffic.
I've aldready verify using a wireshark capture that the template is arriving.
Continously I get:

Can't (yet) decode flowset id 256 from observation domain id 0, because no template to decode it with has been received. This message will usually go away after 1 minute.

Netflow is generated with IXIA packet broker.

Regards
Ana

robcowart commented

Can you share a PCAP of the flows and template? Most like there are field definitions that need to be added to the definitions file used by the decoder.

janniten commented

Hi Rob, Here a capture including a template
Thannk you
ixflow.zip

eijaniee commented

I have the same issue. The collected flow stored into elasticsearch are not the same captured using wireshark.

screen shot 2018-10-17 at 4 16 31 pm

robcowart commented

The fix to decode IXIA IPFIX records is in commit 5f3377e

eijaniee commented

So we just need to upload the file (ipfix.yaml) into the correct path, replacing original file right?

eijaniee commented

I already upload the updated file but still get the same result. Netflow decode error also keep appearing on the screen non stop.

Please advice if I miss any other action.

[WARN ] 2018-10-25 18:36:34.605 [<udp.0] netflow - Can't (yet) decode flowset id 271 from observation domain id 1, because no template to decode it with has been received. This message will usually go away after 1 minute.
[WARN ] 2018-10-25 18:36:34.609 [<udp.0] netflow - Can't (yet) decode flowset id 259 from observation domain id 5, because no template to decode it with has been received. This message will usually go away after 1 minute.
[WARN ] 2018-10-25 18:36:34.609 [<udp.1] netflow - Can't (yet) decode flowset id 256 from observation domain id 0, because no template to decode it with has been received. This message will usually go away after 1 minute.

robcowart commented

Where did you "upload" it to?

The IXIA PCAP that I used did not have flowsets 259 and 271 that you are seeing. You will need to provide a PCAP.

eijaniee commented

Hi,

Here is my pcacp and ipfix.yaml file.

ixia_20181025.pcap.zip

ipfix.yaml.zip

robcowart commented

@eijaniee there are two fields in your flows that were not in the previous sample provided by @janniten. The field IDs are 192 and 193. Do you have any IXIA documentation that describes these (and any other) fields that they might send?

eijaniee commented

I'm still waiting feedback on the reference from ixia guy.

Beside updating the ipfix codec file, do i need to change anything in my logstash config file? Currently here is my logstash config:

input {
udp {
port => 1990
codec => netflow {
versions => [10]
target => ipfix
}
tags => "ixia"
type => ipfix
}
}

output {
if "ixia" in [tags]{
elasticsearch {
hosts => ["x.x.x.x:9200", "x.x.x.x:9200", "x.x.x.x:9200", "x.x.x.x:9200"]
index => "netflow-ixia-%{+YYYY.MM.dd}"
}
}
}

robcowart commented

@eijaniee as your very simple Logstash pipeline is doing no additional processing of the data, there are no changes required.

If you are interested in more advanced enrichment of the IPFIX data, including out-of-the-box dashboards, you might want to try ElastiFlow.

jorritfolmer commented

Closing, fixed in 4.2.0