This repository contains the code of GitHub Actions Runner modified to spawn preemptible GCP instances with Singularity containers and to perform run steps within them.
The software was designed to run in Google Compute Engine. Therefore, it is necessary to prepare some virtual infrastructure prior to installing the runner.
The repositories listed below contain the definitions of the required components:
- github-actions-runner-scalerunner - the image used by preemptible GCP instances that serve as workers (one worker per job).
- github-actions-runner-terraform - a Terraform module used to create the virtual network, firewall rules, cloud NAT and coordinator instance for the runner.
For convenience, an installation script is available that installs dependencies, configures the system, clones the repository and builds the runner.
Start from scratch by following the procedure below:
gcloud compute instances stop instance-2
gcloud compute instances delete instance-2
gcloud compute instances create instance-2 \
--project=catx-ext-umich \
--zone=us-central1-a \
--machine-type=e2-standard-32 \
--network-interface=network-tier=PREMIUM,subnet=default \
--maintenance-policy=MIGRATE \
--service-account=terraform-runner@catx-ext-umich.iam.gserviceaccount.com \
--scopes=https://www.googleapis.com/auth/cloud-platform \
--create-disk=auto-delete=yes,boot=yes,device-name=instance-2,\
image=projects/debian-cloud/global/images/debian-10-buster-v20210916,\
mode=rw,size=256,type=projects/catx-ext-umich/zones/us-central1-a/diskTypes/pd-balanced \
--no-shielded-secure-boot \
--shielded-vtpm \
--shielded-integrity-monitoring \
--reservation-affinity=any
# Delete previous image archive in bucket
# TODO: How to delete uploaded image (Not just in bucket)
gsutil rm gs://catx-ext-umich-worker-bucket/scalenode-9e1d63d.tar.gz
# Connect to refreshed instance
gcloud compute ssh instance-2 --ssh-flag="-ServerAliveInterval=30"
Before running setup.sh
make sure that the following is done:
- Install
git
upon ssh to the created instance - The archive in the bucket and corresponding image is deleted
- Screenshot to-be included
- Fill in the
TOKEN
incoor.sh
- The
TOKEN
can be found here: https://github.com/idea-fasoc/OpenFASOC/settings/actions/runners/new?arch=x64&os=linux underconfigure
. TheTOKEN
expires in a hour.
- The
After everything is setup, run setup.sh
to build everything from scratch. (This script assumes that all virtual infrastructure created by Terraform does not need to be re-configured.)
The manual below assumes that Debian Buster is used to deploy the runner.
The following packages must be installed:
build-essential
- Terraform
sudo apt-get install software-properties-common
curl -fsSL https://apt.releases.hashicorp.com/gpg |
sudo apt-key add -
sudo apt-add-repository "deb [arch=$(dpkg --print-architecture)] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" |
sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
sudo apt-get install apt-transport-https ca-certificates gnupg
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg |
sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
sudo apt-get update && sudo apt-get install google-cloud-sdk
With all prerequisites in place, in order to install the software, follow the steps below:
Install the Google Cloud SDK and setup the project:
# Authenticate with GCP.
gcloud auth login
# Create a GCP project for your runner.
export PROJECT=example-runner-project
gcloud projects create $PROJECT
gcloud config set $PROJECT
# Create and setup a service account.
export SERVICE_ACCOUNT_ID=runner-manager
gcloud iam service-accounts create $SERVICE_ACCOUNT_ID
gcloud projects add-iam-policy-binding $PROJECT \
--member="serviceAccount:$SERVICE_ACCOUNT_ID@$PROJECT.iam.gserviceaccount.com" \
--role="roles/compute.admin"
gcloud projects add-iam-policy-binding $PROJECT \
--member="serviceAccount:$SERVICE_ACCOUNT_ID@$PROJECT.iam.gserviceaccount.com" \
--role="roles/iam.serviceAccountCreator"
gcloud projects add-iam-policy-binding $PROJECT \
--member="serviceAccount:$SERVICE_ACCOUNT_ID@$PROJECT.iam.gserviceaccount.com" \
--role="roles/iam.serviceAccountUser"
# Create and download SA key.
# WARNING: the export below will be used by Terraform later.
# This command is for the OUTPUT_FILE option for "keys create"
export GOOGLE_APPLICATION_CREDENTIALS=~/key.out
gcloud iam service-accounts keys create $GOOGLE_APPLICATION_CREDENTIALS \
--iam-account=$SERVICE_ACCOUNT_ID@$PROJECT.iam.gserviceaccount.com
# Create a GCP bucket for worker image.
export BUCKET=$PROJECT-worker-bucket
gsutil mb gs://$BUCKET
Build and upload the worker image:
# Clone the repository
git clone --recursive https://github.com/antmicro/github-actions-runner-scalerunner.git
cd github-actions-runner-scalerunner
# Compile bzImage
cd buildroot && make BR2_EXTERNAL=../overlay/ scalenode_gcp_defconfig && make
# Prepare a disk for GCP
./make_gcp_image.sh
Save the bucket's IAM policy to a temporary (arbitrary) JSON file
gsutil iam get gs://$BUCKET > /arbitrary/path/file.json
Get the project name and default service account email address. Adjust filter accordingly if a different service account is used
export PROJECT=$(gcloud config get-value project)
export SA=$(gcloud iam service-accounts list --filter=default |
grep -E -o '[a-z0-9._%+-]+@[a-z0-9.-]+(\.[a-z0-9._%+-]+)?[a-z]{2,4}')
Get the absolute path of the Bucket config file
export BUCKET_FILE=/arbitrary/path/file.json
Using the sed
utility to insert required permissions associated with the bucket
sed -i 's/"bindings": \[/"bindings": \[\
{\
"members": \[\
"projectEditor:'"$PROJECT"'",\
"projectOwner:'"$PROJECT"'",\
"serviceAccount:'"$SA"'"\
\],\
"role": "roles\/storage.legacyBucketOwner"\
\},/' $BUCKET_FILE
Upload the modified bucket file back to GCloud
gsutil iam set $BUCKET_FILE gs://$BUCKET
Upload the resulting tar archive
./upload_gcp_image.sh $PROJECT $BUCKET
git clone https://github.com/antmicro/github-actions-runner-terraform.git
terraform init && terraform apply
Connect to the coordinator instance created in the previous step:
gcloud compute ssh gha-runner-coordinator --zone=us-west1-a
Install and configure the runner on the coordinator instance:
# Download and run the installation script.
wget https://raw.githubusercontent.com/antmicro/runner/vm-runners/scripts/install.sh | bash
# The runner software runs as the 'runner' user, so let's sudo into it.
sudo -i -u runner
cd /home/runner/github-actions-runner
# Copy the .vm_specs.json file and adjust the parameters accordingly.
cp .vm_specs.example.json .vm_specs.json
vim .vm_specs.json
# Register the runner in the desired repository.
./config.sh --url https://github.com/$REPOSITORY_ORG/$REPOSITORY_NAME --token $TOKEN --num $SLOTS
In order to start the runners manually, run SCALE=<number of slots> supervisord -n -c supervisord.conf
.
Start the runner by running sudo systemctl start gha-main@$SLOTS
replacing $SLOTS
with the number of runner slots you'd like to allocate.
If you want the software to start automatically, run the command above with the enable
action instead of start
.