peekbank/peekbank-data-import

Create Checklist for Code Review + Add Restraints to Validator

Closed this issue · 2 comments

There are issues that might pop up more often due to schema changes or different interpretations of columns

There is a running document for these points here: https://docs.google.com/document/d/1pZqfS2lt_uFoq4GyvGuzDR7BE0ki5wCDtCXftvv4VEo/edit

Before starting the review process, we might want to reduce the review work needed by extending the validator with all checklist items that can be expressed in code and then running it against all of the datasets

We would need a central runner and a fix for #122, but that would allow us to extend our validator if new things pop up and test all our import scripts against that on the import level.
Not sure how much of this is already happening during the import into the actual database, but pushing the checks forward would make it a lot easier to review the import scripts.

Before starting the review process, we might want to reduce the review work needed by extending the validator with all checklist items that can be expressed in code and then running it against all of the datasets

We would need a central runner and a fix for #122, but that would allow us to extend our validator if new things pop up and test all our import scripts against that on the import level. Not sure how much of this is already happening during the import into the actual database, but pushing the checks forward would make it a lot easier to review the import scripts.

This part is functional now, so we can think about what else to put in the validator