airbnb/streamalert

[improvement] migrate rule "integration" tests out of root tests directory

ryandeivert opened this issue · 1 comments

Background

The rule "integration" tests structure is currently poorly designed, and the tests are embedded in the tests directory (which should be strictly for unit tests). It also makes is prohibitive difficult to create a standalone package/decouple rules+config/do anything in modular way.

Desired Change

Any user rules should have "tests" defined alongside of them. These could follow the convention that most unit-test frameworks use where files are prefixed with test_. The testing framework would then only load files that begin with test_ and end with .json.

Proposed Design

rules/
└── community
    ├── rule_01.py
    └── test_rule_01.json

Things to consider

  • Does the structure of test files themselves need to change?
    • Do we still need the source, etc attributes in the test events? It seems like we
  • Should a given test event could still be able to apply to multiple rules? Or should a file be restricted to testing only rules that it "lives beside". I'm thinking the former, but could be convinced otherwise.
  • How do we handle validate_schema_only test events? Should these be in a "special" folder like `classifier_tests" or handled elsewhere? (this seems like the most problematic part of this change)

fixed with #1179, #1189