Pipelines built with kfp(kubeflow pipeline) execute in k8s cluster.
During the development of kubeflow pipeline components, we usually upload the pipeline to kfp server and run it in k8s, if anything wrong we fix it locally and upload it again.
It should be helpful for the development efficiency if the above process takes place in local host, that's what kfp-local cares.
pip install kfp-local
from kfp_local import LocalClient
local_client = LocalClient()
result = local_client.create_run_from_pipeline_func(
pipeline_func,
arguments={"foo": "bar"},
execution_mode=LocalClient.ExecutionMode("local"),
)
if result.success:
a_output_filepath = result.get_output_file(task_name="a-task", output="a_output_name")
The demo code in Usage executes pipeline in local process with ExecutionMode('local'), ExecutionMode controls how the pipline executes.
There are some options of ExecutionMode:
-
mode: Default execution mode, default 'docker'
-
images_to_exclude: If the image of op is in images_to_exclude, the op is executed in the mode different from default_mode.
-
tasks_to_exclude: If the name of task is in tasks_to_exclude, the op is executed in the mode different from default_mode.
-
docker_options: Docker options used in docker mode, e.g. docker_options=["-e", "foo=bar"].
For more information about how to make use of kfp_local, please refer to unit test.
kfp-local is tested with kfp=1.8.9 for now.
Supports:
- Control flow: Condition, ParallelFor, ExitHandler
- Data passing: InputPath, OutputPath, Input
Don't support for now:
- Importer
- Artifact iterator
- Caching