vertex-ai-project-example
Welcome! This is an example project for demonstrating how to build an extensible, production-ready project on Vertex AI for image use cases. It is built specifically for the Mineral team but has some ideas that might be useful to share.
How to use
Folder structure
- Data labelling
- Data processing
- Training
- Workspace
Workspace
Here is where you create your datasets and training pipelines.
Concepts
- Pipelines as extensible Python classes
- Reusable pipeline factories
- Automated training via Cloud Build
Out-of-scope
MLOps, such as model monitoring, feature store, etc
Setup
Automation
Cloud Build
Cloud Build packages your code and runs it on Google Cloud servers.
Note: Instead of using gcloud
, you can do all the below using the Google Cloud web console UI or even with client libraries.
1. Run all pipelines
The file to determine what commands are run are in the .cloud-build/run_pipelines_cloudbuild.yaml
file.
To run this on the cloud, run the following command:
gcloud builds submit --config=.cloud-build/run_pipelines_cloudbuild.yaml --substitutions=_GCP_BUCKET_NAME="gs://MY_BUCKET_NAME/pipeline_staging"
This will error out as you haven't given the correct permissions to the service account yet. To find out the service account, go the the Cloud Build logs listed in the terminal output for the command and find it under the "EXECUTION DETAILS" tab.
Alternatively, this command may give you the correct one:
gcloud projects get-iam-policy MY_PROJECT | grep cloudbuild
aiplatform.pipelineJobs.create
permission. Grant it access to the roles/aiplatform.user
role which has this permission.
2. The Cloud Build service account needs access to the gcloud projects add-iam-policy-binding MY_PROJECT \
--member='serviceAccount:MY_SERVICE_ACCOUNT@cloudbuild.gserviceaccount.com' \
--role='roles/aiplatform.user'
roles/iam.serviceAccountUser
role:
3. Also grant it the ability to use another service account by giving it the gcloud projects add-iam-policy-binding MY_PROJECT \
--member='serviceAccount:MY_SERVICE_ACCOUNT@cloudbuild.gserviceaccount.com' \
--role='roles/iam.serviceAccountUser'
4. Now submit the build again. This time it should work.
gcloud builds submit --config=.cloud-build/run_pipelines_cloudbuild.yaml --substitutions=_GCP_BUCKET_NAME="gs://MY_BUCKET_NAME/pipeline_staging"
5. (Optional) Set up triggers
Instead of running the Cloud Build command from your local machine, you can set up triggers to automatically run your Cloud Build jobs.
For this repo, you might want to retrain all pipelines on the following conditions:
- Scheduled weekly
- Whenever someone creates a pull-request on your repositoy
- Whenever code is merged to a certain branch
Here are some examples:
1. Create a trigger to run pipelines whenever new code is merged to the 'main' branch:
gcloud beta builds triggers create github --name="run-pipelines" \
--repo-name vertex-ai-project-example \
--repo-owner ivanmkc \
--branch-pattern="^main$" \
--build-config .cloud-build/run_pipelines_cloudbuild.yaml \
--substitutions=_GCP_BUCKET_NAME="gs://MY_BUCKET_NAME/pipeline_staging"
Also see this for more options: https://cloud.google.com/sdk/gcloud/reference/beta/builds/triggers/create/github For more details, see: https://cloud.google.com/build/docs/automating-builds/build-repos-from-github
2. Create a trigger to run pipelines every week on Monday
There doesn't seem to be a way to do this without using the Google Cloud web console UI, so please see instructions here: https://cloud.google.com/build/docs/automating-builds/create-scheduled-triggers