sky-uk/kfp-operator

Add Runconfiguration Artifact definition

Closed this issue · 0 comments

Allow exposing what artifacts a RunConfiguration/RunSchedule/Run can produce. This will allow dependency resolution and dependent triggering.

kind: RunConfiguration
spec:
  run:
    ...
    artifacts:
    - name: a-long-artifact
      path: Pusher:pushed_model:0[pushed == 1]
    - name: a-short-artifact
      path: Pushed:pushed_model
  triggers:
    ...

Extend run completion events to include the resolved artifacts:

{
  "id": "{{ UNIQUE_MESSAGE_ID }}",
  "specversion": "1.0",
  "source": "{{ PROVIDER_NAME }}",
  "type": "org.kubeflow.pipelines.run-completion",
  "datacontenttype": "application/json",
  "data": {
    "status": "succeeded|failed",
    "pipelineName":"{{ PIPELINE_NAME }}",
    "runConfigurationName": "{{ RUNCONFIGURATION_NAMESPACE }}/{{ RUNCONFIGURATION_NAME }}", 
    "runName": "{{ RUN_NAMESPACE }}/{{ RUN_NAME }}", 
    "runId": "{{ RUN_ID }}",
    "servingModelArtifacts": [
      {
        "name":"{{ PIPELINE_NAME }}:{{ WORKFLOW_NAME }}:Pusher:pushed_model:{{ PUSHER_INDEX }}",
        "location":"gs://{{ PIPELINE_ROOT }}/Pusher/pushed_model/{{ MODEL_VERSION }}"
      }
    ],
    "artifacts": [
      {"name": "{{ ARTIFACT_NAME}} ", "location": "{{ ARTIFACT_LOCATION}}"}
    ]
  }
}

Notes:

  • When index is omitted, default to '0'
  • Wildcards are out of scope
  • Consider using https://github.com/hashicorp/go-bexpr for evaluating boolean expression filters
  • servingModelArtifacts in run completion events will be deprecated but not removed

Discarded Options:

kind: RunConfiguration
spec:
  artifacts:
  - name: a-long-artifact
    component: Pusher
    artifact: pushed_model
    filter:
    - key: pushed
      value: 1
  - name: a-short-artifact
    artifact: pushed_model