sky-uk/kfp-operator

Provide fallback pipeline name for scheduled pipeline runs

Opened this issue · 0 comments

jmendesky commented

Overview

Due to kubeflow/pipelines#7274, we cannot infer the name of the pipeline for a completed run that has been created as part of a recurring run. As a result, no event would be emitted (neither failed nor succeeded) for completed recurring runs.

Given that pipelines.kubeflow.org/kfp_sdk_version is not always present, we should now filter workflows based on the presence of pipeline/runid instead.

Possible solutions

  1. Recent versions of MLMD will store lineage of contexts. This means that we can query MLMD for a completed run's pipeline.
  2. Query the KFP API for this metadata.