Nokia IPFIX flowrate is just 3k/s

[rajjaisur]

Hello Jorrit,
I need your help i am receiving around 120k flow per sec.
but I am only able to parse 3k flow per sec.

i am having 24 core(dual processor) processor with 64gb ram physical server dedicated for logstash and elasticsearch only.
it was utilizing almost 85 percent of cpu.

my logstash configuration.

udp {
port  => 4739
codec => netflow{
           versions => 10
            include_flowset_id => true
            cache_save_path => "/root/netflow_template"
            cache_ttl => 999999999
             }
receive_buffer_bytes => 16777216
workers              => 24
id                   => "Netflow_Version_10"
queue_size           => 5000

}

I have already mailed you Pcap and template_cache file.

is there anything you can suggest to increase flow rate.

[robcowart]

You are not anywhere close to the resources that you will require to collect 120K flows/sec. Depending on your exact requirements (retention periods, high-availability, peak vs avg. rates, etc) you will need at least an 8-12 Elasticsearch node cluster (much more for longer retention periods more than a few days), and a similar number of dedicated Logstash nodes.

Even with increased resources, you will need to tune Linux for optimal UDP throughput, as well as other Logstash parameters.

Whether flows, logs or other sources... 120K events per second will require the help of someone with experience dealing with that volume of data.