How to define ILM policy from Ansible ?
astik opened this issue · 7 comments
When adding a beat, after process start, it will load default template into kibana and configure default template and default policy for those templates (metricbeat-* for example).
As ILM is enable by default since 7.0.0, it is quite a surprise to see that index are not rollover every day as it was before.
Policy is by default size 50g and retention 30days. It would be great to be able to define those within ansible.
For now, after policy and template are created, we need to change policy through kibana (or elastic tools) (version 1 becomes version 2), remove the policy from existing index (version 1) and add back policy (version 2).
Hi @astik,
I didn't test it, but you should be able to define your ILM policy in beat_conf variable using Configure index lifecycle management doc:
beat_conf:
...
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "filebeat"
setup.ilm.pattern: "{now/d}-000001"Simple as that !
I can't test it for now, but it looks aligned with what the role will do.
Thanks for the info.
Follow up on this ticket:
- by default, with version 7.6, beat will use respective policy name:
- ilm definition from beat installation does not allow policy definition
- ilm policy definition should be elasticsearch responsability, not beats
- still, ilm policy for beats index is initialized by beat first run with a default policy, wo it may be beat's responsabilty after all for the default one
Default ILM policy are:
filebeat export ilm-policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_age": "30d",
"max_size": "50gb"
}
}
}
}
}
}
metricbeat export ilm-policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_age": "30d",
"max_size": "50gb"
}
}
}
}
}
}
- beat allow the definition of a non existing policy through the use of settings setup.ilm.policy_file (https://www.elastic.co/guide/en/beats/metricbeat/current/ilm.html#setup-ilm-policy_file-option and https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html#setup-ilm-policy_file-option)
- then we could choose to overwrite default policy with the new one through the settings setup.ilm.overwrite (https://www.elastic.co/guide/en/beats/metricbeat/current/ilm.html#setup-ilm-overwrite-option and https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html#setup-ilm-overwrite-option)
any chance to be able, from beat ansible role, to define default ILM policy that is created if none already exist ?
As the beat configuration is expecting a file path, we would need a way to create default policy json configuration file in the ansible-beat role
@jmlrt what do you think of that approach?
FWIW, here is the change i had done to make it work:
- my playbook conf:
- role: elastic.beats
beat: metricbeat
beat_conf:
setup:
dashboards.enabled: true
ilm:
policy_file: /etc/metricbeat/policies/my-metricbeat.json
overwrite: true
metricbeat.modules:
- module: system
metricsets:
- cpu
- load
- ...
enabled: true
period: 10s
processes:
- ".*"
default_ilm_policy: conf/elasticsearch/ilm-policies/my-metricbeat.json
(notice the setup.ilm.policy_file and setup.ilm.overwrite in beat_conf (standard stuff) and the default_ilm_policy parameter (new stuff).
- additions at the end of beats-config.yml:
# Copy default ILM policy file
- name: Create default policies config directory
file:
path: "{{ beats_conf_dir }}/policies"
state: directory
when: default_ilm_policy is defined
- name: Copy default ILM policy file for {{ beat }}
copy:
src: "{{default_ilm_policy}}"
dest: "{{ beats_conf_dir }}/policies/{{default_ilm_policy | basename}}"
when: default_ilm_policy is defined
notify: restart the service
we create a new folder to store default policy (/etc/metricbeat/policies), then we copy our policy to this folder (/etc/metricbeat/policies/my-metricbeat.json).
This is a very naïve approach. It would be better not to have to set up setup.ilm.policy_file and have it automatically set up when default_ilm_policy is defined.
Hi @astik, that make sense.
Being able to copy a file and setup its path in the config file is something that we already do for TLS certs in ansible-elasticsearch (see elastic/ansible-elasticsearch@d7efa20).
Would you be interested to make a pull request for that?
Yes i am =)
I already work on the naive solution past week-end.
Still a WIP as it needs better polishing.