Prefect (Model orchestration tool) learning resources.
- Create conda virtual environment:
conda env create -f requirements.yaml
- Activate the conda environment:
conda activate prefect-env
- Good to go!
- Create a script with the flow. leo.py
- Create a deployment specification for this flow. leo_deployment.py
- Set the Prefect API server URL:
prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api
- Configure storage:
prefect storage create
- Make sure the bucket already exists in S3 buckets.
- Create the deployment:
prefect deployment create leo_deployment.py
- List all of the current deployments:
prefect deployment ls
- Display details of the specific deployment:
prefect deployment inspect 'leonardo_dicaprio_flow/leonardo-deployment'
- Run the deployment locally:
prefect deployment execute leonardo_dicaprio_flow/leonardo-deployment
- Format of the name: flow_name/deployment_name
- Flow may be referenced by multiple deployments, each deployment must have a unique name.
- View the deployment in Prefect UI: Deployments
- Create a work queue:
prefect work-queue create --tag tutorial tutorial_queue
- Note that this command specifically creates a "tutorial" tag on the work queue, meaning the tutorial_queue work queue will only serve deployments that include a "tutorial" tag. This is a good practice to make sure flow runs for a given deployment run only in the correct environments, based on the tags you apply.
- The Prefect API creates the work queue and returns the ID of the queue. Note this ID, you'll use it in a moment to create an agent that polls for work from this queue.
- List of available work queues:
prefect work-queue ls
- Run agent:
prefect agent start 'c86c2f96-f366-498c-acee-d9c8e310ec52'
Remember that:
- The deployment specification included a "tutorial" tag.
- The
tutorial_queue
work queue is defined to serve deployments with a "tutorial" tag. - The agent is configured to pick up work from
tutorial_queue
, so it will only execute flow runs for deployments with a "tutorial" tag.
With a work flow and agent in place, you can create a flow run for leonardo_dicaprio_flow
directly from the UI. Go back to the Prefect UI in your browser and click the Quick Run button next the deployment.
- Update leo_deployment.py with another DeploymentSpec
- Create the deployment:
prefect deployment create leo_deployment.py
- It won't change leonardo_deployment, but it will create a new marvin-deployment.
- Running
leonardo_deployment
logs the message "Hello Leo!". - Running
marvin-deployment
logs the message "Hello Marvin!". marvin-deployment
uses theSubprocessFlowRunner
.
- Running
- It won't change leonardo_deployment, but it will create a new marvin-deployment.
- Both deployments can use the
tutorial_queue
work queue because they have "tutorial" tags. But if you created a new work queue that served deployments with, say, a "dev" tag, onlymarvin-deployment
would be served by that queue and its agents.
- To terminate the agent, simply go to the terminal session where it's running and end the process with either
Ctrl+C
or by terminating the terminal session. - To pause a work queue, run the Prefect CLI command
prefect work-queue pause
, passing the work queue ID. - To delete a work queue, run the command
prefect work-queue delete
, passing the work queue ID. - To terminate the Prefect API server, go to the terminal session where it's running and end the process with either
Ctrl+C
or by terminating the terminal session.