Increase json dictionary value limit from 255 to 346

[AwkwardEquation]

• ansible-rulebook version: 0.10.1
• Python version: 3.8.13
• Operating System: RHEL 7

Attempting to process an incoming webhook from Atlassian Jira, and it exceeds the maximum dictionary value of 255.

I was attempting to receive a webhook from Atlassian Jira for a rulebook, and the webhook utility in Jira sends a json package with too many dictionary values. It appears the max value for ansible-rulebook is 255, but a json package originating from the Jira provided webhook tool is 346.

Received the following message when the webhook hits our ansible-rulebook server while the rulebook is running:
_2023-01-30 09:28:02,479 - asyncio - ERROR - Task exception was never retrieved
future: <Task finished name='Task-8' coro=<RuleSetRunner.drain_source_queue() done, defined at /home/ansible/.local/lib/python3.8/site-packages/ansible_rulebook/engine.py:280> exception=Exception('Only 255 values supported per dictionary found 346')>

This causes the rulebook to error out and stop processing new incoming webhooks.

I experimented by increasing the limit in "ansible_rulebook/util.py" to 350 to see if the rulebook was able to run successfully. I did this by modifying the values in json_count(data) "if len(o) > 255" and " if s> 255:" to 350. This allowed the webhook to be processed successfully without causing the rulebook to exit with an error.

Jira doesn't currently have the ability to limit the json package values using the built in webhook tool. The custom variables provided using a custom webhook is very limited and only provides very basic functionality.

Thanks,
Pat

[mkanoor]

@AwkwardEquation Of the payload that is being sent to the rule engine how many of these variables would be used in rules? There might be a lot of attributes from the payload that can be filtered out. We have a filter that can be used in a rule book to limit the data that is sent into the rule engine. https://github.com/ansible/event-driven-ansible/blob/main/plugins/event_filter/json_filter.py. This filter can be applied per rule book to filter out all the extraneous data and limit the payload to the data that you care about in your rules. The source plugin can be a generic plugin but the filter applied at the rule book can customize the data per rule book.

[AwkwardEquation]

@mkanoor Thank you for your input. I do have filters to process the incoming data to a much more limited set, however, the full size of the json package still comes through and causes the rulebook to error out before those filters are applied.

This is what I'm currently using to filter for example:
filters:
- ansible.eda.json_filter:
include_keys: ['message','reporter','ticket']

These values are for a custom webhook with limited variables, so they don't reflect what is being passed via the full webhook.

[mkanoor]

@AwkwardEquation I have added an example in PR #330. As part of the filter you also need to exclude all other keys with an "*" like shown in the PR. Otherwise the filter includes all the keys and blows past the limit. It's important to keep the payload small and limit it to variables you need in the rules and the ones that you anticipate to use in the playbook or job template.

[AwkwardEquation]

@mkanoor Thank you for your guidance. I was able to process the webhook successfully without overloading ansible-rulebook with too many values. I think this clarification will help others. It might be worthwhile to note the dictionary limit in the documentation in case others run into this situation Thanks again.