Curator 5.8.4 stopped functioning after move to Elasticsearch 7.16.3
jeroenhartgers opened this issue · 5 comments
For usage questions and help
Please create a topic at https://discuss.elastic.co/c/elasticsearch
Perhaps a topic there already has an answer for you!
To submit a bug or report an issue
After moving to Elasticsearch 7.16.3, curator 5.8.4 stopped working altogether.
Running: /usr/local/bin/curator --config /etc/curator/curator.yml /etc/curator/curator-actions.yml
Expected Behavior
Expected cleanup of old indexes as before.
Actual Behavior
Cleanup does not work and a lot of errors and warnings are thrown.
Error on screen:
/usr/local/lib/python3.9/site-packages/elasticsearch/connection/base.py:200: ElasticsearchWarning: this request accesses system indices: [.async-search, .kibana_task_manager_7.16.2_001], but in a future major version, direct access to system indices will be prevented by default warnings.warn(message, category=ElasticsearchWarning)
Error in logs:
2022-01-25 16:22:34,033 INFO Trying Action ID: 12, "delete_indices": Delete indices older than 31 days (based on index name), for tomcat-. 2022-01-25 16:22:34,427 ERROR Failed to complete action: delete_indices. <class 'KeyError'>: 'indices'
Steps to Reproduce the Problem
- Build a docker container with python:3.9-alpine with pip install of curator 5.8.4
- Run
curator_cli --config /etc/curator/curator.yml show-indices --verbose
Specifications
- Version: curator 5.8.4 (running against Elastic 7.16.3)
- Platform: containerized python:3.9-alpine
- Subsystem: k8s 1.22
Context (Environment)
Detailed Description
Was able to fix it by changing file /usr/local/lib/python3.9/site-packages/curator/utils.py
FROM:
client.indices.get_settings(index='_all', params={'expand_wildcards': 'open,closed'})
TO:
client.indices.get_settings(index='a*,b*,c*,d*,e*,f*,g*,h*,i*,j*,k*,l*,m*,n*,o*,p*,q*,r*,s*,t*,u*,v*,w*,x*,y*,z*', params={'expand_wildcards': 'open,closed'})
After this, I was able to see all my indices and cron job that cleans was working as well.
There is probably a cleaner solution than this. In bash I would use [A-Za-z0-9]*, but I could not find it.
hi @jeroenhartgers , do you use the ilm on your indices ?
I've experienced this as well, and able to make it work by adding allow_ilm_indices = True
in the action options.
Thank you @ermusthofa. I do not use ilm on the indices, but I have been playing with it. I will certainly try to add the mentioned action during the next deploy!
Per the above recommendation, I set the allow_ilm_indices
setting and things started working again. I think the problem is here is that curator silently skips indices that have an attached ILM policy. I think it should print out the indices skipped due to this reason with a warning.
I’m sorry you’ve had a hard time, but this has been a default behavior since Elasticsearch 7.0 was released, and it was amply documented at the time.
This is resolved with the release of Curator v7 (and Curator v8)