airbnb/streamalert

Running test command locally should reflect packages configured

jack1902 opened this issue · 1 comments

Background

Description

Currently, if you run the rules_engine locally via ./manage.py test rules this doesn't reflect the additional third party packages listed within conf/lambda.json under rules_engine_config/third_party_libraries. Ideally, the local run should match the packages that will be shipped in the lambda function

Steps to Reproduce (applies to all locally tested 'lambdas')

  • import a package not currently used by the rule_engine (or the classifier, alert_processor etc)
  • install this locally and run ./manage.py test rules - this will work
  • deploy the rules engine ./manage.py deploy -f rule and observe errors of package not found (this is because it wasn't added to the configuration for the lambda)

Desired Change

Packages listed out within the conf files are used as the packages available to the local tester during the stages of the test (classifier packages for classifier test via classifier, rules_engine packages for rules tests via rule and alert_processor packages via live)

Adding update to make this less confusing:

Playbook of how i ran into this

  1. package_name was already installed into my local environment
  2. I added import package_name to a rule within the rules_engine
  3. I ran my local test files via ./manage.py test rule to ensure the rules_engine worked as i expected
  4. I then deployed the rules_engine expecting everything to be cool, it wasn't i had forgotten to update the third_party_libraries piece within the JSON conf file.
  5. I updated this third_party_libraries and re-deployed. Everything worked

What i would have liked to have happened

Running Step 3, i would have liked it to fail with the same error i had ran into in the deployed/zipped up environment in Lambda. Ideally the test command would create seperate virtualenvs to be used by each step:

  • classifier - has a venv containing the packages as stated within the configuration files. This would allow the local tests to fall over if i just imported a package that wasn't explicitly stated within the confguration files.

Same applies to the rules_engine and the alert_processor. As another example, during a live test, it would use each seperate venv to classify the record, then evaluate it through the rules_engine and finally the alert_processor. I am not stating this is an easy thing to achieve as something needs to maintain state outside of each invocation during the local test phases