DB Insights is a tool kit for validating data. It contains premade checks for important data checks. DB Insights provides out-of-the box checks which emit statsd metrics and logs -- alerting you to the health of your database, and the integrity of your data.
DB insights is a single deployable binary that you configure to point at 1 or more databases. Currently it supports the following checks for the following databases:
| Snowflake | Redshift | Postgres | |
|---|---|---|---|
| Freshness | ✅ | ❌ | ✅ |
| Comparisons | ✅ - Primary | ✅ - Secondary | ❌ |
Freshness checks the time of the most recent record within a table.
# examples/freshness.yml
freshness:
targets:
- database: test
schema: public
table: test
column: created_at
- Start postgres locally using docker
$ docker-compose up -d
- Verify that the tset table was created
$ psql -h localhost -U test
Password for user test:
psql (12.4)
Type "help" for help.
test=#
test=# \dt
List of relations
Schema | Name | Type | Owner
--------+------+-------+-------
public | test | table | test
(1 row)
test=# select * from test;
created_at
----------------------------
2021-02-18 19:32:00.519778
(1 row)
- export your connection string
$ export DB_INSIGHTS_CONN_STRING="dbname=test user=test password=test host=localhost sslmode=disable"
- invoke the check
$ go run cmd/dbinsights/main.go -conf=examples/freshness.yml -db=postgres check freshness
hi
DEBU[0000] Freshness Conf: {Targets:[{Database:test Schema:public Table:test Column:created_at}]}
DEBU[0000] postgres.Freshness: executed: "SELECT max(created_at) FROM test.public.test"
DEBU[0000] EmitAll: [{test public test 2021-02-18 19:51:11.490488 +0000 +0000 created_at}]
DEBU[0000] freshness table: test.public.test, total_seconds: 2m43.758708s
Comparisons query 2 different databases and emits metrics about the results of the query. Comparison queries must return
(string, int). All strings must be the same keys.