docker image - flowcoll should recover once elasticsearch become available
archont00 opened this issue · 3 comments
I use a separate container with elasticsearch & kibana.
If a container based on elastiflow/flow-collector:5.0.1 is started before the elasticsearch cluster is up, it logs connection errors:
May 02 12:00:02 hostname flow-collector[835]: {"level":"error","ts":1619949602.9207108,"caller":"elasticsearch/pool.go:44","msg":"unable to put index template into elasticsearch cluster","code":"elasticsearch/invalid-index-template-request","reason":"dial tcp <ip_masked>:9201: connect: connection refused",...
The error continues even after the elasticsearch cluster has started and become available.
To resolve the bug, it is required to restart the flow-collector container.
The problem is quite significant when the server (or docker) is restarted - docker does not provide a reliable way to sort out the container dependencies.
Would it be possible to add a startup script waiting for elasticsearch cluster? Or anything similar.
I just tested this by bringing up the collector, seeing the errors and then bringing up elasticsearch. Once elasticsearch is available it connects fine.
{"level":"error","ts":1619953333.189947,"caller":"elasticsearch/pool.go:44","msg":"unable to put index template into elasticsearch cluster","code":"elasticsearch/invalid-index-template-request","reason":"EOF","stacktrace":"github.com/elastiflow/flowcoll/outputs/elasticsearch.(*Pool).handleElasticsearchIndexTemplate\n\t/Users/rob/go/src/github.com/elastiflow/flowcoll/outputs/elasticsearch/pool.go:44\ngithub.com/elastiflow/flowcoll/outputs/elasticsearch.(*Pool).Run\n\t/Users/rob/go/src/github.com/elastiflow/flowcoll/outputs/elasticsearch/pool.go:22"}
{"level":"error","ts":1619953344.210664,"caller":"elasticsearch/pool.go:44","msg":"unable to put index template into elasticsearch cluster","code":"elasticsearch/invalid-index-template-request","reason":"EOF","stacktrace":"github.com/elastiflow/flowcoll/outputs/elasticsearch.(*Pool).handleElasticsearchIndexTemplate\n\t/Users/rob/go/src/github.com/elastiflow/flowcoll/outputs/elasticsearch/pool.go:44\ngithub.com/elastiflow/flowcoll/outputs/elasticsearch.(*Pool).Run\n\t/Users/rob/go/src/github.com/elastiflow/flowcoll/outputs/elasticsearch/pool.go:22"}
{"level":"info","ts":1619953349.483244,"caller":"elasticsearch/elasticsearch.go:354","msg":"reading index template","source":"elasticsearch","version":"1.1-runtime"}
{"level":"info","ts":1619953349.483318,"caller":"elasticsearch/elasticsearch.go:52","msg":"running"}
BTW, this is the repo for the legacy logstash-based version of ElastiFlow. Please join the community Slack for assistance with the new collector. The rest of the team hangs out there and you are likely to get help more quickly (you got lucky to catch me today).
Okay, moving to Slack.