MarquezProject/marquez

Column lineage query returning null for namespace and name

mattwparas opened this issue · 3 comments

mattwparas commented

It looks like with this change #2821, the column lineage query returns null values, which causes the get dataset endpoint to 500 since namespace and name are marked as non null. I've tested with the old query and it returns the column lineage properly. I have not yet investigated as to why

davidsharp7 commented

Have you got some examples we could look at?

mattwparas commented

Yeah, I can paste some examples - its a bit difficult since we've got quite a bit of data so I'll try to distill it down to something manageable

  • 👍1
sophiely commented

Hi all !

I face the same issue here.
I feel like it's because the namespace and name are provided thanks to the datasets_view but this view only contains the latest version of a dataset.
Sometimes the column lineage is created by an ulterior version so this version won't be find if the datasets_view, I fix the query so as we read in the datasets_version table and not the dataset_views.
Let me know what you think :)