Support pipeline re-runs in Model Update Eventsource

[jmendesky]

Overview

Kubeflow allows re-running of pipeline runs which uses Argo's retry mechanism. This will not create a new workflow resource and instead re-runs pods of the existing resource. In the case when we have already processed this resource, we will miss any subsequent pipeline completions because we will have already marked the workflow as processed.

We should support re-running of pipeline workflows.

Notes

• This can be achieved by clearing the workflows pipelines.kubeflow.org/events-published annotation when moving into the WorkfowRunning state
• We should also consider the case of re-running succeeded pipelines, in which case we may have to consider multiple model artifacts and the order in which they have been produced.