chanzuckerberg/napari-hub

Create data parity workflow for commit activity

manasaV3 opened this issue · 1 comments

manasaV3 commented

Create a workflow for commit activity when we need to update/backfill records for the entire table. This is needed to remove records from dynamo that have been removed in snowflake.

manasaV3 commented

Example use cases:

  • napari-bud-cell-segmenter: It's code_repo has been made private now, I believe. Snowflake table had a rewrite, and after the rewrite, has no data for it. But, the dynamo is surfacing old data that was in Snowflake.

  • bfio: Snowflake had wrong commit data in the past, which still continues to live on in Dynamo. You can see this reflected in the Maintenance timeline for September 2022. The repo's commit history doesn't have any data for that time period. The current data is snowflake reflects the correct commit history for that repo but dynamo doesn't.