logstash is running but not able to see Sflow data
prashant2929 opened this issue · 3 comments
I have installed elastiflow 4.0.1 and logstash is up , i am trying to push sflow data through netflow generator an app provided by solarwinds but i am not able to see data in kibana or logstash nor the index is created
log file
Aug 12 05:51:09 06dbb5745c1c.mylabserver.com logstash[3282]: [2021-08-12T05:51:09,758][INFO ][logstash.filters.geoip ][elastiflow] Using geoip database {:p
ath=>"/etc/logstash/elastiflow-4.0.1/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"}
Aug 12 05:51:10 06dbb5745c1c.mylabserver.com logstash[3282]: [2021-08-12T05:51:10,428][INFO ][logstash.filters.geoip.databasemanager][elastiflow] GeoIP datab
ase path is configured manually so the plugin will not check for update. Keep in mind that if you are not using the database shipped with this plugin, please
go to https://www.maxmind.com/en/geolite2/eula and understand the terms and conditions.
Aug 12 05:51:10 06dbb5745c1c.mylabserver.com logstash[3282]: [2021-08-12T05:51:10,429][INFO ][logstash.filters.geoip ][elastiflow] Using geoip database {:p
ath=>"/etc/logstash/elastiflow-4.0.1/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"}
Aug 12 05:51:31 06dbb5745c1c.mylabserver.com logstash[3282]: [2021-08-12T05:51:31,271][INFO ][logstash.javapipeline ][elastiflow] Starting pipeline {:pipe
line_id=>"elastiflow", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/et
c/logstash/elastiflow-4.0.1/logstash/elastiflow/conf.d/10_input_sflow_ipv4.logstash.conf", "/etc/logstash/elastiflow-4.0.1/logstash/elastiflow/conf.d/20_filt
er_10_begin.logstash.conf", "/etc/logstash/elastiflow-4.0.1/logstash/elastiflow/conf.d/20_filter_40_sflow.logstash.conf", "/etc/logstash/elastiflow-4.0.1/log
stash/elastiflow/conf.d/20_filter_90_post_process.logstash.conf", "/etc/logstash/elastiflow-4.0.1/logstash/elastiflow/conf.d/30_output_10_single.logstash.con
f"], :thread=>"#<Thread:0x52181060 run>"}
Aug 12 05:51:37 06dbb5745c1c.mylabserver.com logstash[3282]: [2021-08-12T05:51:37,899][INFO ][logstash.javapipeline ][elastiflow] Pipeline Java execution
initialization time {"seconds"=>6.62}
Aug 12 05:51:38 06dbb5745c1c.mylabserver.com logstash[3282]: [2021-08-12T05:51:38,047][INFO ][logstash.javapipeline ][elastiflow] Pipeline started {"pipel
ine.id"=>"elastiflow"}
Aug 12 05:51:38 06dbb5745c1c.mylabserver.com logstash[3282]: [2021-08-12T05:51:38,304][INFO ][logstash.agent ] Pipelines running {:count=>1, :runni
ng_pipelines=>[:elastiflow], :non_running_pipelines=>[]}
Aug 12 05:51:38 06dbb5745c1c.mylabserver.com logstash[3282]: [2021-08-12T05:51:38,437][INFO ][logstash.inputs.udp ][elastiflow][b84ce4d636a1c65ebb80b1b6
21fc0a8f4fc9fadf22d8163f472cc7307ca3d5d3] Starting UDP listener {:address=>"0.0.0.0:6343"}
Aug 12 05:51:38 06dbb5745c1c.mylabserver.com logstash[3282]: [2021-08-12T05:51:38,490][WARN ][logstash.inputs.udp ][elastiflow][b84ce4d636a1c65ebb80b1b6
21fc0a8f4fc9fadf22d8163f472cc7307ca3d5d3] Unable to set receive_buffer_bytes to desired size. Requested 33554432 but obtained 212992 bytes.
Aug 12 05:51:38 06dbb5745c1c.mylabserver.com logstash[3282]: [2021-08-12T05:51:38,511][INFO ][logstash.inputs.udp ][elastiflow][b84ce4d636a1c65ebb80b1b6
21fc0a8f4fc9fadf22d8163f472cc7307ca3d5d3] UDP listener started {:address=>"0.0.0.0:6343", :receive_buffer_bytes=>"212992", :queue_size=>"4096"}
As you may have seen in the readme, this project has been deprecated. You should considering using the all new ElastiFlow Unified Flow Collector. More information is available here... https://docs.elastiflow.com/docs/
The Basic license for the new collector is still free and the new collector fixes the many issue with Logstash related to network flow data. You also get many more features and require much less system resources than Logstash.
For assistance with the new collector you can ask questions in the ElastiFlow Community Slack... https://join.slack.com/t/elastiflowcommunity/shared_invite/zt-lv54rhcx-7esE8r8cqggE5mQlShftpA
the support has been called off or even this product or this code wont work for us
The entire focus of the ElastiFlow team and myself is on the new generation of ElastiFlow. We simply have no time to dedicate to this legacy solution. Logstash had a lot of issues processing flow data. Additionally the sFlow codec, which you need, has not been updated by its maintainer for over a year and half.
These issues combined with the poor performance of Logstash, meant it was necessary to move on and build a better collector. This is the new ElastiFlow Unified Flow Collector, which I linked to above.