Curator 5.8.4 stopped functioning after move to Elasticsearch 7.16.3

[jeroenhartgers]

For usage questions and help

Please create a topic at https://discuss.elastic.co/c/elasticsearch

Perhaps a topic there already has an answer for you!

To submit a bug or report an issue

After moving to Elasticsearch 7.16.3, curator 5.8.4 stopped working altogether.

Running: /usr/local/bin/curator --config /etc/curator/curator.yml /etc/curator/curator-actions.yml

Expected Behavior

Expected cleanup of old indexes as before.

Actual Behavior

Cleanup does not work and a lot of errors and warnings are thrown.

Error on screen:
/usr/local/lib/python3.9/site-packages/elasticsearch/connection/base.py:200: ElasticsearchWarning: this request accesses system indices: [.async-search, .kibana_task_manager_7.16.2_001], but in a future major version, direct access to system indices will be prevented by default warnings.warn(message, category=ElasticsearchWarning)

Error in logs:
2022-01-25 16:22:34,033 INFO Trying Action ID: 12, "delete_indices": Delete indices older than 31 days (based on index name), for tomcat-. 2022-01-25 16:22:34,427 ERROR Failed to complete action: delete_indices. <class 'KeyError'>: 'indices'

Steps to Reproduce the Problem

1. Build a docker container with python:3.9-alpine with pip install of curator 5.8.4
2. Run curator_cli --config /etc/curator/curator.yml show-indices --verbose

Specifications

• Version: curator 5.8.4 (running against Elastic 7.16.3)
• Platform: containerized python:3.9-alpine
• Subsystem: k8s 1.22

Context (Environment)

Detailed Description

Was able to fix it by changing file /usr/local/lib/python3.9/site-packages/curator/utils.py
FROM:
client.indices.get_settings(index='_all', params={'expand_wildcards': 'open,closed'})
TO:
client.indices.get_settings(index='a*,b*,c*,d*,e*,f*,g*,h*,i*,j*,k*,l*,m*,n*,o*,p*,q*,r*,s*,t*,u*,v*,w*,x*,y*,z*', params={'expand_wildcards': 'open,closed'})

After this, I was able to see all my indices and cron job that cleans was working as well.

There is probably a cleaner solution than this. In bash I would use [A-Za-z0-9]*, but I could not find it.

[ermusthofa]

hi @jeroenhartgers , do you use the ilm on your indices ?
I've experienced this as well, and able to make it work by adding allow_ilm_indices = True in the action options.

[jeroenhartgers]

Thank you @ermusthofa. I do not use ilm on the indices, but I have been playing with it. I will certainly try to add the mentioned action during the next deploy!

[sabedevops]

Per the above recommendation, I set the allow_ilm_indices setting and things started working again. I think the problem is here is that curator silently skips indices that have an attached ILM policy. I think it should print out the indices skipped due to this reason with a warning.

[untergeek]

I’m sorry you’ve had a hard time, but this has been a default behavior since Elasticsearch 7.0 was released, and it was amply documented at the time.

[untergeek]

This is resolved with the release of Curator v7 (and Curator v8)