cal-itp/data-infra

Deprecate obsolete `staging` tables/views

Opened this issue · 1 comments

As an analytics engineer, I want to deprecate outdated & unused tables and views in our data warehouse to reduce the risk of users accidentally using incorrect or obsolete data and to reduce noise and clutter in audit activities (for example, auditing for models lacking documentation.)

AC:

  • Investigate all models in the staging dataset that aren't being updated regularly (see query below)
    • Check whether they appear in the dbt project (some of these are incremental models)
    • Check whether there is documentation of their intended purpose
  • Deprecate unused staging models
-- query to identify outdated models
SELECT
  table_name, table_type, creation_time
FROM
  `cal-itp-data-infra`.`staging`.INFORMATION_SCHEMA.TABLES
WHERE
  creation_time < "<yesterday's date>"
ORDER BY creation_time

Size is "large" because deprecation process spans multiple weeks