sky-uk/kfp-operator

Manage one-off pipeline runs declaratively

Closed this issue · 0 comments

jmendesky commented

Overview

We want to support managing Kubeflow Pipelines runs with a new Run resource.

Technical notes

  • RunSpec should be immutable (i.e. re-submission will not be allowed)
  • A TTL configuration similar to K8s Job's ttlSecondsAfterFinished should allow the run to be deleted after completion
  • One-off runs should work for both fixed pipeline versions and latest pipeline versions

Implementation Notes

The work can be broken up into the following steps

  • Run with fixed pipeline versions (ignoring updates) #196
  • Implement a Validating Webhook to make the Spec immutable #202
  • Introduce CompletionState to Run's Status field #207
    • Depends on #89
  • Implement TTL #209
  • Allow pipeline reference to be specified without version #210
  • Update docs #212