Add accuracy tests against a known-good dataset
Closed this issue · 1 comments
bmtcril commented
As the schema settles, we should add more testing around making sure the calculations performed in this repo are good / don't accidentally break with future updates. I think the following is a good start, but am open to better ideas:
- Create a fairly substantial dataset using the xapi-db-load tool's CSV output, maybe 100k rows
- Store those in a test folder here, in a folder like
test-seeds
- As part of CI, rename that folder to
seeds
and load those files as seeds to the appropriate tables - Write new dbt tests tagged with
ci
so we can explicitly check the outputs of views and MVs but turn it off by default - In addition to the normal
dbt test
run, we should also run adbt test --select tag:ci
(I'm not 100% sure this will work with default off) and ensure the additional tests all come out correctly - It would be nice to reload the seeds and re-run the tests to check that all of our deduplication is working as well
Where possible it would be great if the tests can be assumed to work outside of the known-good dataset, in which case we should exclude marking them with the ci tag.
saraburns1 commented
examples of queries to test:
A simple one would be something like the fact_enrollments model, which should just have the most recent enrollment status joined with the user data.
The far other end is fact_video_watched_segments which tries to find how many times each 5 second section of a video was watched