Deprecate obsolete `staging` tables/views
Opened this issue · 1 comments
lauriemerrell commented
As an analytics engineer, I want to deprecate outdated & unused tables and views in our data warehouse to reduce the risk of users accidentally using incorrect or obsolete data and to reduce noise and clutter in audit activities (for example, auditing for models lacking documentation.)
AC:
- Investigate all models in the
staging
dataset that aren't being updated regularly (see query below)- Check whether they appear in the dbt project (some of these are incremental models)
- Check whether there is documentation of their intended purpose
- Deprecate unused
staging
models
-- query to identify outdated models
SELECT
table_name, table_type, creation_time
FROM
`cal-itp-data-infra`.`staging`.INFORMATION_SCHEMA.TABLES
WHERE
creation_time < "<yesterday's date>"
ORDER BY creation_time
lauriemerrell commented
Size is "large" because deprecation process spans multiple weeks