geosolutions-it/analytics

Check automatic ElasticSearch index creation

Closed this issue · 5 comments

jemacchi commented

We need to check how it works the automatic index creation in ElasticSearch (when Filebeat output is not ElasticSearch, template creation must be loaded manually. Reference: here ).

lpasquali commented

@jemacchi how did you create indices for unavco?

jemacchi commented

@jemacchi how did you create indices for unavco?

@lpasquali, at the beginning I thought that it was not possible (when using logstash in middle of the process), so created the indexes using an initial reference/call from filebeat (using filebeat console command-line), but then I cleaned indexes (trying steps several times, just to be sure of process) and the indexes where created by logstash without having to create them from filebeat (so far, as I see, what it's not possible is to create indexes TEMPLATEs from logstash)

lpasquali commented

@jemacchi yes this is frustrating in the sense it is quite "political" from elk guys as logstash is not theirs. You can see that there is not one condition with the env option -E output.logstash.enabled=true

I think one has to export the json from filebeat and then apply it through kibana, then apply an index rotation policy.

  • 👍1
lpasquali commented

The two steps are also needed and were part in my mind of a three step implementation in the analitics script see #5

  • 👍1
lpasquali commented

After some test, in unavco, the automatic index creation is not possible for various reasons:

  • the fields created by logstash are different and even importing filebeat's index template by hand, breaks the dashboard.
  • using logstash does not permit automatic index creation directly in filebeat.
    we can't do anything about this for now.

@randomorder @jemacchi

  • 👍1