Should you be able to import dagster jobs as ops?
Closed this issue · 4 comments
Beginner here,
But to me many of the meltano jobs are actually an operation instead.
This would allow you to make a nice meltano: extract op, transform op and load op.
Then join them together nicely in the UI as a job in dagster
Asking because currently the packages just loads a meltano job as an dagster operation, but inside a dagster job. If I have this in my meltano.yml
:
jobs:
- name: fetch_raw_data
tasks:
- utils:raw_data
We will get a job called "fetch_raw_data", with the op called "utils_raw_data"
This is great. However, you are often interested in chaining this with another operation in dagster.
As I understand, only dagster OP can be chained, not jobs.
I have 5 Meltano Jobs I am looking to chain together, such that I can press execute somewhere, and the jobs run in a DAG series.
Closing as you can chain with by using meltano_command_op
example from readme:
from dagster import repository, job
from dagster_meltano import meltano_resource, meltano_run_op
@job(resource_defs={"meltano": meltano_resource})
def meltano_run_job():
tap_done = meltano_run_op("tap-1 target-1")()
meltano_run_op("tap-2 target-2")(tap_done)
@repository()
def repository():
return [meltano_run_job]
You are also able to use multiple tasks. Or doesn't that work for your usecase?
jobs:
- name: fetch_raw_data
tasks:
- utils:raw_data
- another:task
it works brilliantly. Well done.