fluentd-elasticsearch - how to delete old logs
Closed this issue · 1 comments
I am trying to use fluentd as a daemonset in Kubernetes to send logs to Elasticsearch/Kibana 7.5.1.
The problem is the logs end up in elasticsearch as logstash-yyyy.mm.dd, and I can't seem to get a index ILM policy to delete ones older than a certain date. I have tried setting up a index template for indexes with pattern "logstash-*" and alias of "efk-logs" and ILM policy with no success. It gives errors like:
illegal_argument_exception: index name [logstash-2020.02.12] does not match pattern '^.*-\d+$'
Seems similar to this post: https://discuss.elastic.co/t/modify-netflow-index-creation-name/173185
Do I need to set something different in fluentd settings? For example add a suffix of -1 or something? Are there any instructions on how to set this up from start to finish with fluentd? I see the instructions on https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html which cover an index, but the instructions don't seem to apply to using a format like logstash-yyyy.mm.dd
So far I have just been using the instructions here: https://docs.fluentd.org/v/0.12/container-deployment/kubernetes and the official helm charts for Elasticsearch and Kibana. I think I should switch to this helm chart maybe, but I am a bit confused on why the fluentd documentation doesn't mention using this helm chart.
I notice there is also this project, but it seems to also diverge from what is being done here. Although it looks like it has a setting for ILM. https://github.com/uken/fluent-plugin-elasticsearch/blob/master/README.md#enable_ilm
Sorry, wrong place to ask.
I guess you will have more luck here: https://github.com/elastic/kibana