dbt-data-reliability: A Python repository from crabio

Data observability for analytics & data engineers

Monitor your data quality, operation and performance directly from your dbt project.

To learn more, refer to our main repo » | Demo »

Quick start

Add to your packages.yml according to your dbt version:

For dbt >1.3.0:

packages:
  - package: elementary-data/elementary
    version: 0.8.3
    ## Docs: https://docs.elementary-data.com

For dbt >=1.2.0, <1.3.0:

packages:
  - package: elementary-data/elementary
    version: 0.8.3
    ## Docs: https://docs.elementary-data.com

    ## !! Important !! For dbt >=1.2.0 \<1.3.0 ##
    ## (Prevents dbt_utils versions exceptions) ##
  - package: dbt-labs/dbt_utils
    version: [">=0.8.0", "<1.0.0"]

For dbt >=1.0.0, <1.2.0:

packages:
  - package: elementary-data/elementary
    version: 0.8.3
    ## Docs: https://docs.elementary-data.com

    ## !! Important !! For dbt <1.2.0 ##
    ## (Prevents dbt_utils versions exceptions) ##
  - package: dbt-labs/dbt_utils
    version: [">=0.8.0", "<0.9.0"]

After adding to packages.yml and running dbt deps, add to your dbt_project.yml:

models:

## elementary models will be created in the schema '<your_schema>_elementary'
## for details, see docs: https://docs.elementary-data.com/ 
  elementary:
    +schema: 'elementary'

And run dbt run --select elementary.

Check out the full documentation for generating the UI, alerts and adding anomaly detection tests.

Run Results and dbt artifacts

The package automatically uploads the dbt artifacts and run results to tables in your data warehouse:

Run results tables:

dbt_run_results
model_run_results
snapshot_run_results
dbt_invocations
elementary_test_results (all dbt test results)

Metadata tables:

dbt_models
dbt_tests
dbt_sources
dbt_exposures
dbt_metrics
dbt_snapshots

Here you can find additional details about the tables.

Data anomalies detection as dbt tests

Elementary dbt tests collect metrics and metadata over time, such as freshness, volume, schema changes, distribution, cardinality, etc. Executed as any other dbt tests, the Elementary tests alert on anomalies and outliers.

Elementary tests are configured and executed like native tests in your project!

Example of Elementary test config in properties.yml:

models:
  - name: your_model_name
    config:
      elementary:
        timestamp_column: updated_at
    tests:
        - elementary.table_anomalies
        - elementary.all_columns_anomalies

Data observability report

Slack alerts

How it works?

Elementary dbt package creates tables of metadata and test results in your data warehouse, as part of your dbt runs. The CLI tool reads the data from these tables, and is used to generate the UI and alerts.