[improvement] migrate rule "integration" tests out of root tests directory
ryandeivert opened this issue · 1 comments
ryandeivert commented
Background
The rule "integration" tests structure is currently poorly designed, and the tests are embedded in the tests
directory (which should be strictly for unit tests). It also makes is prohibitive difficult to create a standalone package/decouple rules+config/do anything in modular way.
Desired Change
Any user rules should have "tests" defined alongside of them. These could follow the convention that most unit-test frameworks use where files are prefixed with test_
. The testing framework would then only load files that begin with test_
and end with .json
.
Proposed Design
rules/
└── community
├── rule_01.py
└── test_rule_01.json
Things to consider
- Does the structure of test files themselves need to change?
- Do we still need the
source
, etc attributes in the test events? It seems like we
- Do we still need the
- Should a given test event could still be able to apply to multiple rules? Or should a file be restricted to testing only rules that it "lives beside". I'm thinking the former, but could be convinced otherwise.
- How do we handle
validate_schema_only
test events? Should these be in a "special" folder like `classifier_tests" or handled elsewhere? (this seems like the most problematic part of this change)
ryandeivert commented