The run_job_template action is not passing the host limit correctly

Question

The run_job_template action is not passing the host limit correctly

AdamMack2007 opened this issue 2 years ago · 6 comments

AdamMack2007 commented 2 years ago

Please confirm the following

I agree to follow this project's code of conduct.
I have checked the current issues for duplicates.
I understand that ansible-rulebook is open source software provided for free and that I might not receive a timely response.

Bug Summary

When a condition is triggered, the run_job_template action is not successfully passing the limit variable to Automation Controller.

The log output from EDA looks correct, as seen in the 'Actual Results' log output, but Controller shows the limit as 'Limit: {{ event.payload.meta.hosts }}'

Environment

version = '0.13.0'
Executable location = /opt/app-root/bin/ansible-rulebook
Drools_jpy version = 0.3.1
Java home = /usr/lib/jvm/java-17-openjdk
Java version = 17.0.7
Python version = 3.9.16 (main, Dec 21 2022, 10:57:18) [GCC 8.5.0 20210514 (Red Hat 8.5.0-17)]

Steps to reproduce

Testing via curl

curl -X POST http://192.168.6.109:5000/endpoint -d '{"message": "BGP neighbor down","meta": {"hosts": "leaf3"}}' -H 'Content-Type: application/json'

Actual results

EDA log output

2023-05-18 17:03:06,051 - ansible_rulebook.engine - INFO - ruleset define: {"name": "Example", "hosts": ["all"], "sources": [{"EventSource": {"name": "ansible.eda.webhook", "source_name": "ansible.eda.webhook", "source_args": {"host": "0.0.0.0", "port": 5000}, "source_filters": []}}], "rules": [{"Rule": {"name": "Get device info if BGP neighbor down", "condition": {"AllCondition": [{"EqualsExpression": {"lhs": {"Event": "payload.message"}, "rhs": {"String": "BGP neighbor down"}}}]}, "actions": [{"Action": {"action": "run_job_template", "action_args": {"name": "Parsers", "job_args": {"limit": "{{ event.payload.meta.hosts }}"}, "organization": "MackNet"}}}], "enabled": true}}]}

2023-05-18 17:03:06,070 - ansible_rulebook.engine - INFO - load source

2023-05-18 17:03:07,047 - ansible_rulebook.engine - INFO - load source filters

2023-05-18 17:03:07,047 - ansible_rulebook.engine - INFO - loading eda.builtin.insert_meta_info

2023-05-18 17:03:08,189 - ansible_rulebook.engine - INFO - Calling main in ansible.eda.webhook

2023-05-18 17:03:08,194 - ansible_rulebook.websocket - INFO - websocket wss://host.containers.internal/api/eda/ws/ansible-rulebook connecting

2023-05-18 17:03:08,202 - ansible_rulebook.engine - INFO - Waiting for all ruleset tasks to end

2023-05-18 17:03:08 204 [drools-async-evaluator-thread] INFO org.drools.ansible.rulebook.integration.api.io.RuleExecutorChannel - Async channel connected

2023-05-18 17:03:08,249 - ansible_rulebook.rule_set_runner - INFO - Waiting for actions on events from Example

2023-05-18 17:03:08,249 - ansible_rulebook.rule_set_runner - INFO - Waiting for events, ruleset: Example

2023-05-18 17:03:08,285 - ansible_rulebook.websocket - INFO - websocket wss://host.containers.internal/api/eda/ws/ansible-rulebook connected

2023-05-18 17:03:42,874 - aiohttp.access - INFO - 10.0.2.100 [18/May/2023:17:03:42 +0000] "POST /endpoint HTTP/1.1" 200 158 "-" "curl/7.87.0"

2023-05-18 17:03:43 011 [main] INFO org.drools.ansible.rulebook.integration.api.rulesengine.RegisterOnlyAgendaFilter - Activation of effective rule "Get device info if BGP neighbor down" with facts: {m={payload={meta={hosts=leaf3}, message=BGP neighbor down}, meta={headers={Accept=*/*, User-Agent=curl/7.87.0, Host=192.168.6.109:5000, Content-Length=59, Content-Type=application/json}, endpoint=endpoint, received_at=2023-05-18T17:03:42.870982Z, source={name=ansible.eda.webhook, type=ansible.eda.webhook}, uuid=a1cf2a8c-2af9-45e6-99c6-eb5e4560b059}}}

2023-05-18 17:03:43,034 - ansible_rulebook.rule_generator - INFO - calling Get device info if BGP neighbor down

2023-05-18 17:03:43,036 - ansible_rulebook.rule_set_runner - INFO - call_action run_job_template

2023-05-18 17:03:43,038 - ansible_rulebook.rule_set_runner - INFO - substitute_variables [{'name': 'Parsers', 'job_args': {'limit': '{{ event.payload.meta.hosts }}'}, 'organization': 'MackNet'}] [{'event': {'payload': {'meta': {'hosts': 'leaf3'}, 'message': 'BGP neighbor down'}, 'meta': {'headers': {'Accept': '*/*', 'User-Agent': 'curl/7.87.0', 'Host': '192.168.6.109:5000', 'Content-Length': '59', 'Content-Type': 'application/json'}, 'endpoint': 'endpoint', 'received_at': '2023-05-18T17:03:42.870982Z', 'source': {'name': 'ansible.eda.webhook', 'type': 'ansible.eda.webhook'}, 'uuid': 'a1cf2a8c-2af9-45e6-99c6-eb5e4560b059'}}}]

2023-05-18 17:03:43,054 - ansible_rulebook.rule_set_runner - INFO - action args: {'name': 'Parsers', 'job_args': {'limit': 'leaf3'}, 'organization': 'MackNet'}

2023-05-18 17:03:43,054 - ansible_rulebook.builtin - INFO - running job template: Parsers, organization: MackNet

2023-05-18 17:03:43,055 - ansible_rulebook.builtin - INFO - ruleset: Example, rule Get device info if BGP neighbor down

Controller limit setting:

Expected results

Automation controller shows the correct host limit and not the variable string.

Additional information

No response

Answer 1 · 2023-05-18T17:34:22.000Z

I think the problem is not in the job_template action but in the webhook plugin because the webhook plugin is putting the payload in a nested field in the event.
ansible-rulebook expects to find meta at the first level. In event.meta not in event.payload.meta

The webhook plugin is not able to receive it directly.

The way to do it is using the insert_hosts_to_meta filter.
https://github.com/ansible/event-driven-ansible/blob/main/extensions/eda/plugins/event_filter/insert_hosts_to_meta.py
https://ansible-rulebook.readthedocs.io/en/latest/filters.html

Answer 2 · 2023-05-24T17:01:26.000Z

@Alex-Izquierdo I tried the following but no luck, anything that stands out? There's not a lot of documentation so I am going off of just the python code that you provided and debugging.

- name: Example
  hosts: all
  sources:
    - ansible.eda.webhook:
        host: 0.0.0.0
        port: 5000
      filters:
        - ansible.eda.insert_hosts_to_meta:
            host_path: event.payload.event.fields.source
  rules:
    - name: Get Device info if BGP down
      condition: event.payload.event is defined
      throttle:
        once_within: 5 minutes
        group_by_attributes:
          - event.payload.event.fields.source
      action:
        run_job_template:
          name: Network Info Report
          job_args:
            limit: "{{ event.meta.hosts }}"
          organization: MackNet

Here is the log output from EDA Controller

2023-05-24 17:03:10,645 - ansible_rulebook.rule_set_runner - INFO - substitute_variables [{'name': 'Network Info Report', 'job_args': {'limit': '{{ event.payload.event.fields.source }}'}, 'organization': 'MackNet'}] [{'event': {'payload': {'job_definition_id': '646d24e0444fce6ad8a6366f', 'backlog': [], 'event_definition_type': 'aggregation-v1', 'event_definition_id': '646d026a444fce6ad8a6104a', 'event_definition_description': '', 'event_definition_title': 'Arista BGP Status', 'event': {'timerange_end': None, 'timestamp_processing': '2023-05-24T17:02:53.619Z', 'origin_context': 'urn:graylog:message:es:graylog_4:de032110-fa53-11ed-8029-0242262f0850', 'streams': [], 'source_streams': ['000000000000000000000001'], 'source': 'GrayLog.local', 'message': 'Arista BGP Status', 'priority': 2, 'group_by_fields': {}, 'key_tuple': [], 'alert': True, 'event_definition_type': 'aggregation-v1', 'event_definition_id': '646d026a444fce6ad8a6104a', 'id': '01H17BDA5KFRBF4KW954TXAEG0', 'fields': {'payload': 'leaf1 Rib: %BGP-5-ADJCHANGE: peer 10.0.2.0 (AS 64512) old state Established event Stop new state Idle', 'source': 'leaf1'}, 'key': '', 'timestamp': '2023-05-24T16:56:03.000Z', 'timerange_start': None}, 'job_trigger_id': '646e433d444fce6ad8a771a5'}, 'meta': {'headers': {'Connection': 'Keep-Alive', 'User-Agent': 'okhttp/3.14.6', 'Host': '192.168.6.109:5000', 'Accept-Encoding': 'gzip', 'Content-Length': '957', 'Content-Type': 'application/json'}, 'endpoint': 'endpoint', 'received_at': '2023-05-24T17:02:48.071396Z', 'source': {'name': 'ansible.eda.webhook', 'type': 'ansible.eda.webhook'}, 'uuid': '276bc0fd-193c-41cd-b8d9-c5d4a8c0a0f9'}}}]

2023-05-24 17:03:10,649 - ansible_rulebook.rule_set_runner - INFO - action args: {'name': 'Network Info Report', 'job_args': {'limit': 'leaf1'}, 'organization': 'MackNet'}

Answer 3 · 2023-05-24T17:49:02.000Z

Hi @AdamMack2007 There is no need to pass directly the "limit" arg to the action, ansible-rulebook will do it implicitly based on event.meta.hosts

All looks fine. What it is the exact problem in this new try? We would need to see the whole output to determine what it's happening.

Answer 4 · 2023-05-24T20:45:54.000Z

Here are the issues I'm experiencing:

When the call is performed to EDA Controller, the insert_hosts_to_meta filter doesn't appear to be injecting the host info into meta.

This is my debug output from "event.meta"

{'headers': {'Connection': 'Keep-Alive', 'User-Agent': 'okhttp/3.14.6', 'Host': '192.168.6.109:5000', 'Accept-Encoding': 'gzip', 'Content-Length': '959', 'Content-Type': 'application/json'}, 'endpoint': 'endpoint', 'received_at': '2023-05-24T20:24:31.768761Z', 'source': {'name': 'ansible.eda.webhook', 'type': 'ansible.eda.webhook'}, 'uuid': '7619687a-f8c6-4d6c-9986-4bc40cfb4464'}

With the job_args : {"limit"} section removed, the default action is "limit: all" within the job template as opposed to the targeted hosts, which is likely tied to issue 1

I understand it's supposed to happen behind the scenes, but is there any reason we cannot override that default behavior and inject a variable?

Also, is there a particular log that would help with troubleshooting this?

Answer 5 · 2023-05-25T08:22:37.000Z

The flag -v or -vv for a complete debug of events should be enough. You can override the job params, I said you that hosts are set implicitly just for your information. According to your previous message, the hosts seemed properly set to "leaf1"

Take into account that your Job template must enable the option "prompt on launch" to be able to process the limit arg. Ref: https://stackoverflow.com/questions/71687700/ansible-tower-api-pass-inventory-for-a-job-in-a-post-payload/71804727#71804727

Answer 6 · 2023-12-01T15:07:25.000Z

I think the problem is not in the job_template action but in the webhook plugin because the webhook plugin is putting the payload in a nested field in the event. ansible-rulebook expects to find meta at the first level. In event.meta not in event.payload.meta

The webhook plugin is not able to receive it directly.

The way to do it is using the insert_hosts_to_meta filter. https://github.com/ansible/event-driven-ansible/blob/main/extensions/eda/plugins/event_filter/insert_hosts_to_meta.py https://ansible-rulebook.readthedocs.io/en/latest/filters.html

PLAYBOOK: recieved-event.yml ***************************************************
Positional arguments: recieved-event.yml
verbosity: 4
remote_user: linadmin
connection: smart
timeout: 60
become: True
become_method: sudo
tags: ('all',)
inventory: ('/tmp/edaeoh4gq5c/inventory/default_inventory.yml',)
subset: vbrhelhyb02.orktion.net
extra_vars: ('@/tmp/edaeoh4gq5c/env/extravars',)
forks: 5
1 plays in recieved-event.yml

It's able to extract host info as subset, however it's end up with
skipping: no hosts matched

what is this subset doing here. it's extracted from events.meta

{
"substitute_variables": [
{
"name": "recieved-event.yml",
"verbosity": 4
}
],
"events": {
"m_1": {
"alert": {
"status": "firing",
"labels": {
"alertname": "HighCPUUsage",
"instance": "vbrhelhyb02.orktion.net:9100",
"severity": "warning"
},
"annotations": {
"description": "CPU usage on vbrhelhyb02.orktion.net:9100 is above 60%.",
"summary": "High CPU Usage on vbrhelhyb02.orktion.net:9100"
},
"startsAt": "2023-12-01T08:57:09.622Z",
"endsAt": "0001-01-01T00:00:00Z",
"generatorURL": "http://vbrhelmon01:9090/graph?g0.expr=100+-+%28avg+by+%28instance%29+%28irate%28node_cpu_seconds_total%7Bmode%3D%22idle%22%7D%5B5m%5D%29%29+%2A+100%29+%3E+60&g0.tab=1",
"fingerprint": "7be4b65d565b653a"
},
"meta": {
"endpoint": "alerts",
"headers": {
"Host": "10.10.4.217:8000",
"User-Agent": "Alertmanager/0.26.0",
"Content-Length": "1426",
"Content-Type": "application/json"
},
"hosts": ["vbrhelhyb02.orktion.net"],
"source": {
"name": "ansible.eda.alertmanager",
"type": "ansible.eda.alertmanager"
},
"received_at": "2023-12-01T14:41:10.066055Z",
"uuid": "a5078a55-213e-4178-a3f5-7459eba965b3"
}
}
},
"action_args": {
"name": "recieved-event.yml",
"verbosity": 4
}
}