oliver-twist

DAG Auditor

oliver-twist is a dag auditing tool that audits the DBT DAG and generates a summary report. There is a docs site, including descriptions of all currently implemented rules.

Getting Started

To get started, install the package

$ pip install olivertwist

and then run it by passing it your dbt manifest JSON

olivertwist check manifest.json

This will report any failures to the console, and also in HTML format in a directory called target. You can optionally auto-open the report in a browser with:

olivertwist check manifest.json --browser

You can also tell Oliver to load and run your own custom rules using the --add-rules-from option. See documentation for full details.

Full options are available with:

olivertwist check --help

Configuration

All rules are enabled by default. To change this you need a configuration file called olivertwist.yml in the same directory you are running olivertwist. An example configuration is shown below:

version: '1.0'
universal:
  - id: no-rejoin-models
    enabled: false
  - id: no-disabled-models
    enabled: true

There is a command to help you generate the config automatically:

olivertwist config

This will show all the available rules and allow you to toggle the ones that you want to enforce.

Local Development

Clone this repo and install the project:

poetry install

Install pre-commit hooks for linting

This is optional, but highly recommended to avoid annoying linting failure in CI.

poetry run pre-commit install

To run the pre-commit hooks locally:

poetry run pre-commit run

To get the latest versions of the dependencies and to update the poetry.lock file run:

poetry update

To run oliver-twist and generate the summary report run:

poetry run olivertwist example_manifest.json

Working with diagrams

To update and regenerate the images that illustrate rule failures in the documentation follow the next steps:

update the diagrams using the mermaid syntax
install yarn
cd docs/diagrams
./generate.sh
inspect the generated images in ./docs/diagrams/output/
if you're happy with the results, run ./copy.sh so that they are copied over to ./docs/images
you can now reference those images. e.g. in .docs/rules.md

Creating a distribution

poetry build --format wheel

octoenergy/oliver-twist