Every keyword in test is timeing out after 5 sec due to --verbose mode

[marcaurele]

Short description

• OPA version: 0.58.0
• Executing: docker run ... opa test --verbose results in a timeout exit after 5s
• Executing: docker run ... opa test works fine
• Linux environment, local debian developer station

I also run the test with a matching pattern -r ... to only grab the test I was working on but I believe it does not affect the every issue.

Steps To Reproduce

Code

The list of "actions" entries in all_allowed_actions must be much larger, we have about hundreds of those ones.

# policy.rego
package authorization

role_manager := "MANAGER"
role_owner := "OWNER"
role_editor := "EDITOR"
role_reader := "READER"

ROLES_ALL := {role_reader, role_editor, role_manager, role_owner}
ROLES_FROM_EDITOR := {role_editor, role_manager, role_owner}
ROLES_FROM_OWNER := {role_owner}

all_allowed_actions := {
	"cloud:project:v3:projects:create": ROLES_FROM_EDITOR,
	"cloud:project:v3:projects:depth-data-delete": ROLES_FROM_EDITOR,
	"cloud:project:v3:projects:depth-data-list": ROLES_ALL,
	"cloud:project:v3:projects:destroy": ROLES_FROM_EDITOR,
	"cumulus:account:add-credits": ROLES_FROM_OWNER,
}

allow {
	allowed_roles := all_allowed_actions[input.action]
	role := input.roles[input.path[_]]
	allowed_roles[role]
}

# test.rego
package authorization_test
import future.keywords.every

actions_allowed_for_editor := {
		"cloud:project:v3:projects:create",
		"cloud:project:v3:projects:depth-data-delete",
		"cloud:project:v3:projects:depth-data-list",
		"cloud:project:v3:projects:destroy",
}

test_editor_permitted {
    every action in actions_allowed_for_editor {
        input_data := {
            "action": action,
            "roles": {
			"0000-0000-0004": "MEMBER",
			"0000-0000-0003": "READER",
			"0000-0000-0002": "EDITOR",
			"0000-0000-0001": "MANAGER",
			"0000-0000-0000": "OWNER",
		},
            "path": ["0000-0000-0005", "0006-0000-0003", "0000-0000-0002"]
        }
        data.authorization.allow with input as input_data
    }
}

Expected behavior

The test function should run much faster, without timing out because of those long lists.

Additional context

When I run these test with hundreds of entries in actions_allowed_for_editor the test is reaching the 5s timeout and gets kill:

SUMMARY
--------------------------------------------------------------------------------
/test/mab_test.rego:
data.authorization_test.test.test_mab_permitted: ERROR (5.001104217s)
  eval_cancel_error: caller cancelled query execution
data.authorization_test.test.test_mab_not_permitted: PASS (78.127613ms)

If the test suite is run without the --verbose option, it works fine.

[ashutosh-narkar]

Thanks for the report @marcaurele. We'll need to look into this. When you turn on verbose mode, you enable query tracing and given the large test case could result in longer runs. If this is the case, then the --timeout flag should help. Not sure if this is the issue. Have you tried to run the test w/o using every?

@johanfylling any ideas?

[marcaurele]

@ashutosh-narkar no, I haven't run without every as this is the recommend way to implement a sort of parametrized test in #2176 .

I can make the tests pass by increasing the timeout value. Currently I've set it to 30 seconds, but for example it could need to be revisited after a while depending on how many more entries does the every loop has to tackle. Not very scalable in a sense.

[stale]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days. Although currently inactive, the issue could still be considered and actively worked on in the future. More details about the use-case this issue attempts to address, the value provided by completing it or possible solutions to resolve it would help to prioritize the issue.