Enforce metric data parity between Snowflake and DynamoDb
Opened this issue · 0 comments
manasaV3 commented
As the hub is not fetching data directly from GitHub/ PyPi and is relying on the snowflake data, we treat snowflake data as the single source of truth, and the data in dynamo should maintain parity with snowflake.
In the cases where data gets added/removed from snowflake tables/views that the hub consumes from, for any reason, we should be able to update the hub data to reflect that. We need an automated workflow to validate that our data is in sync with Snowflake and heal the discrepancies.
This workflow could even be set to run periodically to ensure data quality.