quantile-development/dagster-meltano

Should you be able to import dagster jobs as ops?

Closed this issue · 4 comments

ReneTC commented

Beginner here,
But to me many of the meltano jobs are actually an operation instead.

This would allow you to make a nice meltano: extract op, transform op and load op.
Then join them together nicely in the UI as a job in dagster

ReneTC commented

Asking because currently the packages just loads a meltano job as an dagster operation, but inside a dagster job. If I have this in my meltano.yml:

jobs:
- name: fetch_raw_data
  tasks:
  - utils:raw_data

We will get a job called "fetch_raw_data", with the op called "utils_raw_data"
image

This is great. However, you are often interested in chaining this with another operation in dagster.

As I understand, only dagster OP can be chained, not jobs.

I have 5 Meltano Jobs I am looking to chain together, such that I can press execute somewhere, and the jobs run in a DAG series.

ReneTC commented

Closing as you can chain with by using meltano_command_op

example from readme:

from dagster import repository, job
from dagster_meltano import meltano_resource, meltano_run_op

@job(resource_defs={"meltano": meltano_resource})
def meltano_run_job():
    tap_done = meltano_run_op("tap-1 target-1")()
    meltano_run_op("tap-2 target-2")(tap_done)

@repository()
def repository():
    return [meltano_run_job]

You are also able to use multiple tasks. Or doesn't that work for your usecase?

jobs:
- name: fetch_raw_data
  tasks:
  - utils:raw_data
  - another:task
ReneTC commented

it works brilliantly. Well done.