openedx/aspects-dbt

Add accuracy tests against a known-good dataset

Closed this issue · 1 comments

As the schema settles, we should add more testing around making sure the calculations performed in this repo are good / don't accidentally break with future updates. I think the following is a good start, but am open to better ideas:

  • Create a fairly substantial dataset using the xapi-db-load tool's CSV output, maybe 100k rows
  • Store those in a test folder here, in a folder like test-seeds
  • As part of CI, rename that folder to seeds and load those files as seeds to the appropriate tables
  • Write new dbt tests tagged with ci so we can explicitly check the outputs of views and MVs but turn it off by default
  • In addition to the normal dbt test run, we should also run a dbt test --select tag:ci (I'm not 100% sure this will work with default off) and ensure the additional tests all come out correctly
  • It would be nice to reload the seeds and re-run the tests to check that all of our deduplication is working as well

Where possible it would be great if the tests can be assumed to work outside of the known-good dataset, in which case we should exclude marking them with the ci tag.

examples of queries to test:
A simple one would be something like the fact_enrollments model, which should just have the most recent enrollment status joined with the user data.
The far other end is fact_video_watched_segments which tries to find how many times each 5 second section of a video was watched