MarquezProject/marquez

Streaming jobs do not cumulate datasets sent through a run

pawel-big-lebowski opened this issue · 0 comments

I observed a streaming job that was correctly emitting OL event with input and output datasets which lead to a nice lineage graph. However, a single fail event at the end (which did not have any input / output datasets) broke the graph.

TODO:

  • write an integration test which emits streaming job OL event with inputs/outputs
  • emit OL event for the same run with no input / output datasets
  • make sure API responds with lineage connecting the datasets