/demeter

Build a REST API on top of your data warehouse

Primary LanguageTypeScriptApache License 2.0Apache-2.0

Demeter

The dbt Metrics API

Demeter turns your dbt project into a metrics platform. You get a REST/GraphQL API so you can query, build interactive apps, and use your metrics from anywhere.

View live demo

Deploy

About

Demeter gives you a REST API (including a GraphQL endpoint) to query your dbt metrics

graphql

Installation

Prerequisites

  1. A dbt project with metrics
  2. Node

Quickstart (local)

  1. 📦 Install dbt-metrics-api
    • Add the following to your dbt project's packages.yml file:
    - git: "https://github.com/mjirv/demeter.git"
      revision: main
    • Run dbt deps
  2. ✅ Run
    • Run cd dbt_packages/dbt_metrics_api/server && npm i && node dist/
    • Navigate to http://localhost:3001/graphql in your browser to start querying your API!

Usage

Routes

dbt-metrics-api has 4 routes:

GET /metrics

  • returns a JSON array of your project's metrics

  • supports "name", "type", "model", and "package_name" query strings for filtering

    $ curl -X GET "http://localhost:3001/metrics?type=count&package_name=jaffle_shop"
    
    [{"unique_id":"metric.jaffle_shop.orders","package_name":"jaffle_shop","model":"ref('orders')","name":"orders","description":"The number of orders","label":"Orders","type":"count","filters":[],"time_grains":["day","week","month","quarter","year"],"dimensions":["status","customer_id"]},{"unique_id":"metric.jaffle_shop.orders2","package_name":"jaffle_shop","model":"ref('orders')","name":"orders2","description":"The number of orders","label":"Orders","type":"count","filters":[],"time_grains":["day","week","month","quarter","year"],"dimensions":["status","customer_id"]}]
    

GET /metrics/:metric_name

  • returns a JSON object with keys unique_id, package_name, model, name, description, label, type, filters, time_grains, dimensions

    $ curl -X GET "http://localhost:3001/metrics/orders"
    
    {"unique_id":"metric.jaffle_shop.orders","package_name":"jaffle_shop","model":"ref('orders')","name":"orders","description":"The number of orders","label":"Orders","type":"count","filters":[],"time_grains":["day","week","month","quarter","year"],"dimensions":["status","customer_id"]}
    

POST /metrics/:metric_name

  • Queries a metric and returns the result

  • Accepts a JSON object in the request body with the following properties: grain, dimensions, start_date, end_date (start_date and end_date are optional)

  • Returns a JSON object or CSV depending on your Accept: header (application/json or text/csv)

    $ curl http://localhost:3001/metrics/orders -H "Content-Type: application/json" -H "Accept: application/json" -d '{"grain": "year", "start_date": "2017-01-01", "end_date": "2019-01-01"}'
    
    [{"period": "2017-01-01", "orders": 0.0}, {"period": "2018-01-01", "orders": 99.0}, {"period": "2019-01-01", "orders": 0.0}]
    

POST /graphql

Advanced

Running a production instance

On your favorite cloud hosting service (e.g. Heroku or GCP Cloud Run) install this repo via the GitHub URL.

Look in .example.env for the available environment variables and configure them as needed.

Authentication

dbt-metrics-api supports Kable for authentication.

To get started:

  1. Sign up at https://kable.io (the free tier includes unlimited test requests and up to 10,000 live requests)

  2. Note your client ID and secret and add them to your server/.env.local file:

    KABLE_CLIENT_ID=
    KABLE_CLIENT_SECRET=
    
    # you will need to set to LIVE in production
    KABLE_ENV=TEST
  3. Requests will now require authentication with a customer ID and secret that you set up in your Kable dashboard using X-CLIENT-ID and X-API-KEY headers

    $ curl -X GET "http://localhost:3001/metrics/orders" -H "X-CLIENT-ID: test-customer-1" -H "X-API-KEY: sk_test.some.secret.key"