[Re]Some problem inside my Elasticstack + Suricata
ainayves opened this issue · 13 comments
ainayves commented
I solved my last problem, but now, different problems appear in the logs of Elastic and Kibana
Elastic log error
`
aina@elasticsearch:~$ sudo head /var/log/elasticsearch/elasticsearch.log
[2021-07-28T13:22:52,898][WARN ][o.e.x.m.e.l.LocalExporter] [eY6v6GM] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:125) ~[x-pack-monitoring-6.8.0.jar:6.8.0]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
**Kibana log error**
Jul 28 15:44:48 kibana kibana[1530]: {"type":"log","@timestamp":"2021-07-28T12:44:48Z","tags":["error","task_manager"],"pid":1530,"message":"Failed to poll for work: [cluster_block_exception] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {"path":"/.kibana_task_manager/_doc/Maps-maps_telemetry/_update","query":{"if_seq_no":11,"if_primary_term":2,"refresh":"true"},"body":"{\"doc\":{\"type\":\"task\",\"task\":{\"taskType\":\"maps_telemetry\",\"state\":\"{\\\"runs\\\":1,\\\"stats\\\":{\\\"mapsTotalCount\\\":0,\\\"timeCaptured\\\":\\\"2021-07-12T10:00:18.993Z\\\",\\\"attributesPerMap\\\":{\\\"dataSourcesCount\\\":{\\\"min\\\":0,\\\"max\\\":0,\\\"avg\\\":0},\\\"layersCount\\\":{\\\"min\\\":0,\\\"max\\\":0,\\\"avg\\\":0},\\\"layerTypesCount\\\":{},\\\"emsVectorLayersCount\\\":{}}}}\",\"params\":\"{}\",\"attempts\":0,\"scheduledAt\":\"2021-07-12T10:00:14.897Z\",\"runAt\":\"2021-07-28T12:45:48.369Z\",\"status\":\"running\"},\"kibana\":{\"uuid\":\"c9ffff37-0cdd-43c4-b95c-ca38ea93aee8\",\"version\":6080399,\"apiVersion\":1}}}","statusCode":403,"response":"{\"error\":{\"root_cause\":[{\"type\":\"cluster_block_exception\",\"reason\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}],\"type\":\"cluster_block_exception\",\"reason\":\"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403}"}"}
`
Logstash seems to be working :
`
Jul 28 14:49:20 logstash logstash[2264]: [2021-07-28T14:49:20,449][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/synlite_suricata/geoipdbs/GeoLite2-City.mmdb"}
Jul 28 14:49:20 logstash logstash[2264]: [2021-07-28T14:49:20,767][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/synlite-suricata_stats-1.0.1
Jul 28 14:49:20 logstash logstash[2264]: [2021-07-28T14:49:20,769][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/synlite-suricata-1.0.1
Jul 28 14:49:20 logstash logstash[2264]: [2021-07-28T14:49:20,882][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/etc/logstash/synlite_suricata/geoipdbs/GeoLite2-ASN.mmdb"}
`
And suricata is working well with fielbeat
I am using ELK 6.8.0
Could you help me please???
ainayves commented
My elasticsearch.yml
network.host: 0.0.0.0
http.port: 9200
indices.query.bool.max_clause_count: 8192
search.max_buckets: 250000
My kibana.yml
server.port: 5601
server.host: "192.168.56.108"
elasticsearch.hosts: ["http://192.168.56.103:9200"]
ainayves commented
The error on Kibana GUI
robcowart commented
Can you send the Filebeat logs?
ainayves commented
Can you send the Filebeat logs?
Filebeat logs
`
2021-07-28T16:47:58.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8190,"time":{"ms":1}},"total":{"ticks":189750,"time":{"ms":6},"value":189750},"user":{"ticks":181560,"time":{"ms":5}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66330116}},"memstats":{"gc_next":160857440,"memory_alloc":81089160,"memory_total":937463480,"rss":-163840}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.08,"15":1.18,"5":1.18,"norm":{"1":1.08,"15":1.18,"5":1.18}}}}}}
`
robcowart commented
Is that the entire log? It doesn't look like it.
ainayves commented
2021-07-28T16:35:28.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8130},"total":{"ticks":187830,"time":{"ms":8},"value":187830},"user":{"ticks":179700,"time":{"ms":8}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65580115}},"memstats":{"gc_next":160860992,"memory_alloc":80998232,"memory_total":930295944}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.1,"15":1.19,"5":1.2,"norm":{"1":1.1,"15":1.19,"5":1.2}}}}}}
2021-07-28T16:35:49.590+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:35:49.590+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1455 reconnect attempt(s)
2021-07-28T16:35:49.591+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:35:49.591+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:35:49.591+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:35:49.591+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:35:58.408+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8130,"time":{"ms":1}},"total":{"ticks":187840,"time":{"ms":6},"value":187840},"user":{"ticks":179710,"time":{"ms":5}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65610118}},"memstats":{"gc_next":160860992,"memory_alloc":81445208,"memory_total":930742920}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.16,"15":1.2,"5":1.21,"norm":{"1":1.16,"15":1.2,"5":1.21}}}}}}
2021-07-28T16:36:28.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8140,"time":{"ms":6}},"total":{"ticks":188150,"time":{"ms":304},"value":188150},"user":{"ticks":180010,"time":{"ms":298}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65640117}},"memstats":{"gc_next":160857440,"memory_alloc":80590152,"memory_total":931018080}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.11,"15":1.19,"5":1.19,"norm":{"1":1.11,"15":1.19,"5":1.19}}}}}}
2021-07-28T16:36:38.481+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:36:38.481+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1456 reconnect attempt(s)
2021-07-28T16:36:38.481+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:36:38.481+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:36:38.481+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:36:38.481+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:36:58.408+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8140,"time":{"ms":5}},"total":{"ticks":188150,"time":{"ms":8},"value":188150},"user":{"ticks":180010,"time":{"ms":3}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65670115}},"memstats":{"gc_next":160857440,"memory_alloc":80719032,"memory_total":931146960}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.33,"15":1.21,"5":1.24,"norm":{"1":1.33,"15":1.21,"5":1.24}}}}}}
2021-07-28T16:37:28.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8140,"time":{"ms":3}},"total":{"ticks":188150,"time":{"ms":6},"value":188150},"user":{"ticks":180010,"time":{"ms":3}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65700115}},"memstats":{"gc_next":160857440,"memory_alloc":81005320,"memory_total":931433248}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.88,"15":1.17,"5":1.13,"norm":{"1":0.88,"15":1.17,"5":1.13}}}}}}
2021-07-28T16:37:35.711+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:37:35.711+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1457 reconnect attempt(s)
2021-07-28T16:37:35.712+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:37:35.712+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:37:35.712+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:37:35.712+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:37:58.409+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8150,"time":{"ms":1}},"total":{"ticks":188170,"time":{"ms":8},"value":188170},"user":{"ticks":180020,"time":{"ms":7}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65730115}},"memstats":{"gc_next":160857440,"memory_alloc":81288824,"memory_total":931716752}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":904}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.84,"15":1.16,"5":1.1,"norm":{"1":0.84,"15":1.16,"5":1.1}}}}}}
2021-07-28T16:38:26.127+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:38:26.127+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1458 reconnect attempt(s)
2021-07-28T16:38:26.127+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:38:26.128+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:38:26.128+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:38:26.128+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:38:28.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8150},"total":{"ticks":188480,"time":{"ms":307},"value":188480},"user":{"ticks":180330,"time":{"ms":307}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65760115}},"memstats":{"gc_next":160858432,"memory_alloc":80435952,"memory_total":931998384}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.19,"15":1.18,"5":1.16,"norm":{"1":1.19,"15":1.18,"5":1.16}}}}}}
2021-07-28T16:38:58.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8150},"total":{"ticks":188480,"time":{"ms":5},"value":188480},"user":{"ticks":180330,"time":{"ms":5}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65790115}},"memstats":{"gc_next":160858432,"memory_alloc":80720696,"memory_total":932283128}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.41,"15":1.2,"5":1.21,"norm":{"1":1.41,"15":1.2,"5":1.21}}}}}}
2021-07-28T16:39:14.245+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:39:14.245+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1459 reconnect attempt(s)
2021-07-28T16:39:14.245+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:39:14.246+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:39:14.246+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:39:14.246+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:39:28.409+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8150,"time":{"ms":8}},"total":{"ticks":188480,"time":{"ms":8},"value":188480},"user":{"ticks":180330}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65820115}},"memstats":{"gc_next":160858432,"memory_alloc":81004616,"memory_total":932567048}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.4,"15":1.2,"5":1.22,"norm":{"1":1.4,"15":1.2,"5":1.22}}}}}}
2021-07-28T16:39:58.408+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8160,"time":{"ms":4}},"total":{"ticks":188500,"time":{"ms":6},"value":188500},"user":{"ticks":180340,"time":{"ms":2}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65850115}},"memstats":{"gc_next":160858432,"memory_alloc":81282088,"memory_total":932844520}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.23,"15":1.19,"5":1.2,"norm":{"1":1.23,"15":1.19,"5":1.2}}}}}}
2021-07-28T16:40:13.300+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:40:13.300+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1460 reconnect attempt(s)
2021-07-28T16:40:13.300+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:40:13.300+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:40:13.301+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:40:13.301+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:40:28.607+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8170,"time":{"ms":11}},"total":{"ticks":188780,"time":{"ms":289},"value":188780},"user":{"ticks":180610,"time":{"ms":278}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65880318}},"memstats":{"gc_next":160871744,"memory_alloc":81566072,"memory_total":933128504}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":904}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.25,"15":1.19,"5":1.2,"norm":{"1":1.25,"15":1.19,"5":1.2}}}}}}
2021-07-28T16:40:58.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8170,"time":{"ms":3}},"total":{"ticks":188800,"time":{"ms":16},"value":188800},"user":{"ticks":180630,"time":{"ms":13}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65910116}},"memstats":{"gc_next":160871744,"memory_alloc":80866888,"memory_total":933560080}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.19,"15":1.18,"5":1.18,"norm":{"1":1.19,"15":1.18,"5":1.18}}}}}}
2021-07-28T16:41:02.569+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:41:02.569+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1461 reconnect attempt(s)
2021-07-28T16:41:02.570+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:41:02.570+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:41:02.570+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:41:02.570+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:41:28.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8170,"time":{"ms":1}},"total":{"ticks":188800,"time":{"ms":8},"value":188800},"user":{"ticks":180630,"time":{"ms":7}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65940115}},"memstats":{"gc_next":160871744,"memory_alloc":80996824,"memory_total":933690016}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.18,"15":1.18,"5":1.18,"norm":{"1":1.18,"15":1.18,"5":1.18}}}}}}
2021-07-28T16:41:43.008+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:41:43.009+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1462 reconnect attempt(s)
2021-07-28T16:41:43.009+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:41:43.009+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:41:43.009+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:41:43.009+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:41:58.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8170},"total":{"ticks":188810,"time":{"ms":6},"value":188810},"user":{"ticks":180640,"time":{"ms":6}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":65970118}},"memstats":{"gc_next":160871744,"memory_alloc":81280360,"memory_total":933973552}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.29,"15":1.19,"5":1.21,"norm":{"1":1.29,"15":1.19,"5":1.21}}}}}}
2021-07-28T16:42:22.864+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:42:22.864+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1463 reconnect attempt(s)
2021-07-28T16:42:22.864+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:42:22.864+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:42:22.864+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:42:22.864+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:42:28.408+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8170,"time":{"ms":1}},"total":{"ticks":188820,"time":{"ms":7},"value":188820},"user":{"ticks":180650,"time":{"ms":6}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66000116}},"memstats":{"gc_next":160871744,"memory_alloc":81572360,"memory_total":934265552,"rss":270336}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":904}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.47,"15":1.21,"5":1.26,"norm":{"1":1.47,"15":1.21,"5":1.26}}}}}}
2021-07-28T16:42:58.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8180,"time":{"ms":2}},"total":{"ticks":189130,"time":{"ms":303},"value":189130},"user":{"ticks":180950,"time":{"ms":301}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66030116}},"memstats":{"gc_next":160860960,"memory_alloc":80523288,"memory_total":934622168}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.23,"15":1.2,"5":1.23,"norm":{"1":1.23,"15":1.2,"5":1.23}}}}}}
2021-07-28T16:42:58.535+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:42:58.535+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1464 reconnect attempt(s)
2021-07-28T16:42:58.535+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:42:58.535+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:42:58.535+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:42:58.535+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:43:28.408+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8180},"total":{"ticks":189130,"time":{"ms":7},"value":189130},"user":{"ticks":180950,"time":{"ms":7}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66060115}},"memstats":{"gc_next":160860960,"memory_alloc":80734808,"memory_total":934833688}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.18,"15":1.2,"5":1.22,"norm":{"1":1.18,"15":1.2,"5":1.22}}}}}}
2021-07-28T16:43:40.284+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:43:40.285+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1465 reconnect attempt(s)
2021-07-28T16:43:40.285+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:43:40.285+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:43:40.285+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:43:40.285+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:43:58.412+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8180,"time":{"ms":1}},"total":{"ticks":189140,"time":{"ms":8},"value":189140},"user":{"ticks":180960,"time":{"ms":7}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66090119}},"memstats":{"gc_next":160860960,"memory_alloc":81018376,"memory_total":935117256}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.84,"15":1.17,"5":1.14,"norm":{"1":0.84,"15":1.17,"5":1.14}}}}}}
2021-07-28T16:44:13.279+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:44:13.279+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1466 reconnect attempt(s)
2021-07-28T16:44:13.280+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:44:13.280+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:44:13.280+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:44:13.280+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:44:28.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8180},"total":{"ticks":189150,"time":{"ms":7},"value":189150},"user":{"ticks":180970,"time":{"ms":7}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66120116}},"memstats":{"gc_next":160860960,"memory_alloc":81378696,"memory_total":935477576}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":904}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.04,"15":1.18,"5":1.16,"norm":{"1":1.04,"15":1.18,"5":1.16}}}}}}
2021-07-28T16:44:54.692+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:44:54.692+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1467 reconnect attempt(s)
2021-07-28T16:44:54.692+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:44:54.692+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:44:54.692+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:44:54.692+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:44:58.408+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8180,"time":{"ms":9}},"total":{"ticks":189420,"time":{"ms":283},"value":189420},"user":{"ticks":181240,"time":{"ms":274}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66150117}},"memstats":{"gc_next":160857440,"memory_alloc":80528296,"memory_total":935765232}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.28,"15":1.19,"5":1.21,"norm":{"1":1.28,"15":1.19,"5":1.21}}}}}}
2021-07-28T16:45:28.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8190,"time":{"ms":2}},"total":{"ticks":189430,"time":{"ms":4},"value":189430},"user":{"ticks":181240,"time":{"ms":2}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66180118}},"memstats":{"gc_next":160857440,"memory_alloc":80731704,"memory_total":935968640}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.27,"15":1.2,"5":1.22,"norm":{"1":1.27,"15":1.2,"5":1.22}}}}}}
2021-07-28T16:45:33.707+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:45:33.707+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1468 reconnect attempt(s)
2021-07-28T16:45:33.708+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:45:33.708+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:45:33.708+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:45:33.708+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:45:58.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8190,"time":{"ms":1}},"total":{"ticks":189440,"time":{"ms":7},"value":189440},"user":{"ticks":181250,"time":{"ms":6}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66210115}},"memstats":{"gc_next":160857440,"memory_alloc":81013480,"memory_total":936250416}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.02,"15":1.18,"5":1.16,"norm":{"1":1.02,"15":1.18,"5":1.16}}}}}}
2021-07-28T16:46:22.414+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:46:22.415+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1469 reconnect attempt(s)
2021-07-28T16:46:22.416+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:46:22.416+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:46:22.416+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:46:22.416+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:46:28.411+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8190},"total":{"ticks":189450,"time":{"ms":7},"value":189450},"user":{"ticks":181260,"time":{"ms":7}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66240117}},"memstats":{"gc_next":160857440,"memory_alloc":81300088,"memory_total":936537024}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":904}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.23,"15":1.19,"5":1.2,"norm":{"1":1.23,"15":1.19,"5":1.2}}}}}}
2021-07-28T16:46:58.408+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8190},"total":{"ticks":189740,"time":{"ms":295},"value":189740},"user":{"ticks":181550,"time":{"ms":295}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66270115}},"memstats":{"gc_next":160857440,"memory_alloc":80436368,"memory_total":936810688}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.34,"15":1.2,"5":1.23,"norm":{"1":1.34,"15":1.2,"5":1.23}}}}}}
2021-07-28T16:47:14.244+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:47:14.244+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1470 reconnect attempt(s)
2021-07-28T16:47:14.244+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:47:14.244+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:47:14.244+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:47:14.244+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:47:28.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8190,"time":{"ms":4}},"total":{"ticks":189740,"time":{"ms":8},"value":189740},"user":{"ticks":181550,"time":{"ms":4}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66300118}},"memstats":{"gc_next":160857440,"memory_alloc":80728616,"memory_total":937102936}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.34,"15":1.2,"5":1.24,"norm":{"1":1.34,"15":1.2,"5":1.24}}}}}}
2021-07-28T16:47:58.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8190,"time":{"ms":1}},"total":{"ticks":189750,"time":{"ms":6},"value":189750},"user":{"ticks":181560,"time":{"ms":5}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66330116}},"memstats":{"gc_next":160857440,"memory_alloc":81089160,"memory_total":937463480,"rss":-163840}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.08,"15":1.18,"5":1.18,"norm":{"1":1.08,"15":1.18,"5":1.18}}}}}}
2021-07-28T16:48:04.507+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:48:04.507+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1471 reconnect attempt(s)
2021-07-28T16:48:04.508+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:48:04.508+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:48:04.508+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:48:04.508+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:48:28.409+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8200,"time":{"ms":3}},"total":{"ticks":189760,"time":{"ms":7},"value":189760},"user":{"ticks":181560,"time":{"ms":4}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66360115}},"memstats":{"gc_next":160857440,"memory_alloc":81295896,"memory_total":937670216}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.99,"15":1.17,"5":1.15,"norm":{"1":0.99,"15":1.17,"5":1.15}}}}}}
2021-07-28T16:48:36.902+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:48:36.902+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1472 reconnect attempt(s)
2021-07-28T16:48:36.902+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:48:36.902+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:48:36.903+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:48:36.903+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:48:58.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8200,"time":{"ms":4}},"total":{"ticks":190060,"time":{"ms":302},"value":190060},"user":{"ticks":181860,"time":{"ms":298}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66390116}},"memstats":{"gc_next":160860992,"memory_alloc":80510888,"memory_total":938021760}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":904}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.04,"15":1.17,"5":1.14,"norm":{"1":1.04,"15":1.17,"5":1.14}}}}}}
2021-07-28T16:49:28.408+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8200,"time":{"ms":3}},"total":{"ticks":190060,"time":{"ms":5},"value":190060},"user":{"ticks":181860,"time":{"ms":2}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66420115}},"memstats":{"gc_next":160860992,"memory_alloc":80724136,"memory_total":938235008}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.94,"15":1.16,"5":1.11,"norm":{"1":0.94,"15":1.16,"5":1.11}}}}}}
2021-07-28T16:49:36.143+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:49:36.144+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1473 reconnect attempt(s)
2021-07-28T16:49:36.144+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:49:36.144+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:49:36.144+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:49:36.144+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:49:58.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8210,"time":{"ms":5}},"total":{"ticks":190080,"time":{"ms":8},"value":190080},"user":{"ticks":181870,"time":{"ms":3}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66450115}},"memstats":{"gc_next":160860992,"memory_alloc":81002376,"memory_total":938513248}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.24,"15":1.18,"5":1.16,"norm":{"1":1.24,"15":1.18,"5":1.16}}}}}}
2021-07-28T16:50:23.317+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:50:23.317+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1474 reconnect attempt(s)
2021-07-28T16:50:23.318+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:50:23.318+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:50:23.318+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:50:23.318+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:50:28.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8210,"time":{"ms":6}},"total":{"ticks":190080,"time":{"ms":6},"value":190080},"user":{"ticks":181870}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66480115}},"memstats":{"gc_next":160860992,"memory_alloc":81295336,"memory_total":938806208,"rss":-32768}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.37,"15":1.19,"5":1.19,"norm":{"1":1.37,"15":1.19,"5":1.19}}}}}}
2021-07-28T16:50:58.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8220,"time":{"ms":1}},"total":{"ticks":190360,"time":{"ms":276},"value":190360},"user":{"ticks":182140,"time":{"ms":275}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66510115}},"memstats":{"gc_next":160857440,"memory_alloc":80435936,"memory_total":939080592}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.14,"15":1.17,"5":1.15,"norm":{"1":1.14,"15":1.17,"5":1.15}}}}}}
2021-07-28T16:50:58.433+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:50:58.434+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1475 reconnect attempt(s)
2021-07-28T16:50:58.435+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:50:58.437+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:50:58.437+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:50:58.438+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:51:28.407+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8220,"time":{"ms":3}},"total":{"ticks":190370,"time":{"ms":6},"value":190370},"user":{"ticks":182150,"time":{"ms":3}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66540115}},"memstats":{"gc_next":160857440,"memory_alloc":80728600,"memory_total":939373256}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":904}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.34,"15":1.19,"5":1.2,"norm":{"1":1.34,"15":1.19,"5":1.2}}}}}}
2021-07-28T16:51:43.820+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:51:43.820+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1476 reconnect attempt(s)
2021-07-28T16:51:43.820+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:51:43.821+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:51:43.821+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:51:43.821+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:58:46.307+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8220,"time":{"ms":5}},"total":{"ticks":190370,"time":{"ms":7},"value":190370},"user":{"ticks":182150,"time":{"ms":2}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66570116}},"memstats":{"gc_next":160857440,"memory_alloc":81167496,"memory_total":939812152}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":931}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.44,"15":1.2,"5":1.24,"norm":{"1":1.44,"15":1.2,"5":1.24}}}}}}
2021-07-28T16:59:16.308+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8230,"time":{"ms":4}},"total":{"ticks":190380,"time":{"ms":6},"value":190380},"user":{"ticks":182150,"time":{"ms":2}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66600115}},"memstats":{"gc_next":160857440,"memory_alloc":81287080,"memory_total":939931736}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.55,"15":1.22,"5":1.28,"norm":{"1":1.55,"15":1.22,"5":1.28}}}}}}
2021-07-28T16:59:30.978+0300 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://192.168.56.109:5044)): dial tcp 192.168.56.109:5044: connect: connection refused
2021-07-28T16:59:30.978+0300 INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://192.168.56.109:5044)) with 1477 reconnect attempt(s)
2021-07-28T16:59:30.978+0300 INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2021-07-28T16:59:30.978+0300 INFO [publish] pipeline/retry.go:191 done
2021-07-28T16:59:30.978+0300 INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2021-07-28T16:59:30.978+0300 INFO [publish] pipeline/retry.go:168 done
2021-07-28T16:59:46.307+0300 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8230,"time":{"ms":4}},"total":{"ticks":190670,"time":{"ms":297},"value":190670},"user":{"ticks":182440,"time":{"ms":293}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":8},"info":{"ephemeral_id":"6287a44d-4618-4c68-b7d7-1d1eb339a31d","uptime":{"ms":66630115}},"memstats":{"gc_next":160857408,"memory_alloc":80439184,"memory_total":940220744}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":3,"events":{"active":4118,"retry":423}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":1.3,"15":1.21,"5":1.25,"norm":{"1":1.3,"15":1.21,"5":1.25}}}}}}
ainayves commented
192.168.56.109:5044 is logstash endpoint
robcowart commented
You need to figure out why it can't connect to Logstash.
ainayves commented
You mean elasticsearch??? or filebeat???
robcowart commented
Filebeat. You can see the error in the Filebeat logs.
ainayves commented
So why : when I am doing a TCPDUMP on 5044 port in logstash ,, I am getting packets??
`sudo tcpdump -i enp0s8 -s 1500 port 5044
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s8, link-type EN10MB (Ethernet), capture size 1500 bytes
17:08:10.130365 IP 192.168.56.107.44528 > 192.168.56.109.5044: Flags [S], seq 2887510672, win 64240, options [mss 1460,sackOK,TS val 1683559313 ecr 0,nop,wscale 7], length 0
17:08:10.130389 IP 192.168.56.109.5044 > 192.168.56.107.44528: Flags [R.], seq 0, ack 2887510673, win 0, length 0
`
192.168.56.107 is my suricata +filebeat server
robcowart commented
Just because packets are arriving doesn't mean that they a being received. Look at the logs... the details matter. It says "connection refused". It doesn't say "timed-out", or "no response"... it says the connection was REFUSED. Could be firewall, selinux, apparmor or any number of other things about your environment that are preventing a connection. That is what you have to figure out.
robcowart commented
Closing all issues as this project has been archived.