- The module
- Prerequisites
- Usage
- Examples
- Requirements
- Providers
- Modules
- Resources
- Inputs
- Outputs
- Contributors ✨
This Terraform modules creates a GitLab CI runner. A blog post describes the original version of the the runner. See the post at 040code. The original setup of the module is based on the blog post: Auto scale GitLab CI runners and save 90% on EC2 costs.
The runners created by the module use spot instances by default for running the builds using the docker+machine
executor.
- Shared cache in S3 with life cycle management to clear objects after x days.
- Logs streamed to CloudWatch.
- Runner agents registered automatically.
The name of the runner agent and runner is set with the overrides variable. Adding an agent runner name tag does not work.
...
overrides = {
name_sg = ""
name_runner_agent_instance = "Gitlab Runner Agent"
name_docker_machine_runners = "Gitlab Runner Terraform"
}
//this doesn't work
agent_tags = merge(local.my_tags, map("Name", "Gitlab Runner Agent"))
The runner supports 3 main scenarios:
In this scenario the runner agent is running on a single EC2 node and runners are created by docker machine using spot instances. Runners will scale automatically based on the configuration. The module creates a S3 cache by default, which is shared across runners (spot instances).
In this scenario the multiple runner agents can be created with different configuration by instantiating the module multiple times. Runners will scale automatically based on the configuration. The S3 cache can be shared across runners by managing the cache outside of the module.
In this scenario not docker machine is used but docker to schedule the builds. Builds will run on the same EC2 instance as the agent. No auto scaling is supported.
Ensure you have Terraform installed. The modules is based on Terraform 0.11, see .terraform-version
for the used version. A handy tool to mange your Terraform version is tfenv.
On macOS it is simple to install tfenv
using brew
.
brew install tfenv
Next install a Terraform version.
tfenv install <version>
Ensure you have setup your AWS credentials. The module requires access to IAM, EC2, CloudWatch, S3 and SSM.
In order to be able to destroy the module, you will need to run from a host with both jq
and aws
installed and accessible in the environment.
On macOS it is simple to install them using brew
.
brew install jq awscli
The GitLab runner EC2 instance requires the following service linked roles:
- AWSServiceRoleForAutoScaling
- AWSServiceRoleForEC2Spot
By default the EC2 instance is allowed to create the required roles, but this can be disabled by setting the option allow_iam_service_linked_role_creation
to false
. If disabled you must ensure the roles exist. You can create them manually or via Terraform.
resource "aws_iam_service_linked_role" "spot" {
aws_service_name = "spot.amazonaws.com"
}
resource "aws_iam_service_linked_role" "autoscaling" {
aws_service_name = "autoscaling.amazonaws.com"
}
By default the runner is registered on initial deployment. In previous versions of this module this was a manual process. The manual process is still supported but will be removed in future releases. The runner token will be stored in the AWS SSM parameter store. See example for more details.
To register the runner automatically set the variable gitlab_runner_registration_config["registration_token"]
. This token value can be found in your GitLab project, group, or global settings. For a generic runner you can find the token in the admin section. By default the runner will be locked to the target project, not run untagged. Below is an example of the configuration map.
gitlab_runner_registration_config = {
registration_token = "<registration token>"
tag_list = "<your tags, comma separated>"
description = "<some description>"
locked_to_project = "true"
run_untagged = "false"
maximum_timeout = "3600"
access_level = "<not_protected OR ref_protected, ref_protected runner will only run on pipelines triggered on protected branches. Defaults to not_protected>"
}
For migration to the new setup simply add the runner token to the parameter store. Once the runner is started it will lookup the required values via the parameter store. If the value is null
a new runner will be registered and a new token created/stored.
# set the following variables, look up the variables in your Terraform config.
# see your Terraform variables to fill in the vars below.
aws-region=<${var.aws_region}>
token=<runner-token-see-your-gitlab-runner>
parameter-name=<${var.environment}>-<${var.secure_parameter_store_runner_token_key}>
aws ssm put-parameter --overwrite --type SecureString --name "${parameter-name}" --value ${token} --region "${aws-region}"
Once you have created the parameter, you must remove the variable runners_token
from your config. The next time your GitLab runner instance is created it will look up the token from the SSM parameter store.
Finally, the runner still supports the manual runner creation. No changes are required. Please keep in mind that this setup will be removed in future releases.
A few option are provided to access the runner instance:
- Provide a EC2 key pair to access the runner by setting ``.
- Access via the Session Manager (SSM) by setting
enable_runner_ssm_access
totrue
. The policy to allow access via SSM is not very restrictive. - By setting none of the above, no keys or extra policies will be attached to the instance. You can still configure you own policies by attaching them to
runner_agent_role_arn
.
By default the module creates a a cache for the runner in S3. Old objects are automatically removed via a configurable life cycle policy on the bucket.
Creation of the bucket can be disabled and managed outside this module. A good use case is for sharing the cache across multiple runners. For this purpose the cache is implemented as a sub module. For more details see the cache module. An example implementation of this use case can be found in the runner-public example.
We set up a build pipeline for 13 developers (plus one technical user running Renovate to update all
dependencies) in 3 availability zones. Running 2 different type of machines. One for cloud deployment,
aka terraform apply
(t3.medium, on demand, 3 idles, 15 max, 45 minutes idle time). The other one
for all other jobs (c5.xlarge, spot, 9 idles, 60 max, 45 minutes idle time). All machines are
using a shared S3 cache. The pipeline is running from 7am till 7pm. In the non peak hours all
docker+machine instances are killed.
Per day costs: $20. We are quite satisfied with this pipeline.
Update the variables in terraform.tfvars
according to your needs and add the following variables. See the previous step for instructions on how to obtain the token.
runner_name = "NAME_OF_YOUR_RUNNER"
gitlab_url = "GITLAB_URL"
runner_token = "RUNNER_TOKEN"
The base image used to host the GitLab Runner agent is the latest available Amazon Linux 2 HVM EBS AMI. In previous versions of this module a hard coded list of AMIs per region was provided. This list has been replaced by a search filter to find the latest AMI. Setting the filter to amzn2-ami-hvm-2.0.20200207.1-x86_64-ebs
will allow you to version lock the target AMI.
Below is a basic examples of usages of the module. Regarding the dependencies such as a VPC, have a look at the default example.
module "runner" {
# https://registry.terraform.io/modules/npalm/gitlab-runner/aws/
source = "npalm/gitlab-runner/aws"
aws_region = "eu-west-1"
environment = "spot-runners"
vpc_id = module.vpc.vpc_id
subnet_ids_gitlab_runner = module.vpc.private_subnets
subnet_id_runners = element(module.vpc.private_subnets, 0)
runners_name = "docker-default"
runners_gitlab_url = "https://gitlab.com"
gitlab_runner_registration_config = {
registration_token = "my-token"
tag_list = "docker"
description = "runner default"
locked_to_project = "true"
run_untagged = "false"
maximum_timeout = "3600"
}
}
A few examples are provided. Use the following steps to deploy. Ensure your AWS and Terraform environment is set up correctly. All commands below should be run from the terraform-aws-gitlab-runner/examples/<example-dir>
directory.
The version of Terraform is locked down via tfenv, see the .terraform-version
file for the expected versions. Providers are locked down as well in the providers.tf
file.
The examples are configured with defaults that should work in general. The examples are in general configured for the region Ireland eu-west-1
. The only parameter that needs to be provided is the GitLab registration token. The token can be found in GitLab in the runner section (global, group or repo scope). Create a file terrafrom.tfvars
and the registration token.
registration_token = "MY_TOKEN"
Run terraform init
to initialize Terraform. Next you can run terraform plan
to inspect the resources that will be created.
To create the runner, run:
terraform apply
To destroy the runner, run:
terraform destroy
Name | Version |
---|---|
terraform | >= 0.13.0 |
aws | >= 3.35.0 |
Name | Version |
---|---|
aws | >= 3.35.0 |
null | n/a |
Name | Source | Version |
---|---|---|
cache | ./modules/cache | n/a |
Name | Description | Type | Default | Required |
---|---|---|---|---|
agent_tags | Map of tags that will be added to agent EC2 instances. | map(string) |
{} |
no |
allow_iam_service_linked_role_creation | Boolean used to control attaching the policy to a runner instance to create service linked roles. | bool |
true |
no |
ami_filter | List of maps used to create the AMI filter for the Gitlab runner agent AMI. Must resolve to an Amazon Linux 1 or 2 image. | map(list(string)) |
{ |
no |
ami_owners | The list of owners used to select the AMI of Gitlab runner agent instances. | list(string) |
[ |
no |
arn_format | ARN format to be used. May be changed to support deployment in GovCloud/China regions. | string |
"arn:aws" |
no |
asg_delete_timeout | Timeout when trying to delete the Runner ASG. | string |
"10m" |
no |
asg_max_instance_lifetime | The seconds before an instance is refreshed in the ASG. | number |
null |
no |
aws_region | AWS region. | string |
n/a | yes |
cache_bucket | Configuration to control the creation of the cache bucket. By default the bucket will be created and used as shared cache. To use the same cache across multiple runners disable the creation of the cache and provide a policy and bucket name. See the public runner example for more details. | map(any) |
{ |
no |
cache_bucket_name_include_account_id | Boolean to add current account ID to cache bucket name. | bool |
true |
no |
cache_bucket_prefix | Prefix for s3 cache bucket name. | string |
"" |
no |
cache_bucket_set_random_suffix | Append the cache bucket name with a random string suffix | bool |
false |
no |
cache_bucket_versioning | Boolean used to enable versioning on the cache bucket, false by default. | bool |
false |
no |
cache_expiration_days | Number of days before cache objects expires. | number |
1 |
no |
cache_shared | Enables cache sharing between runners, false by default. | bool |
false |
no |
cloudwatch_logging_retention_in_days | Retention for cloudwatch logs. Defaults to unlimited | number |
0 |
no |
docker_machine_download_url | Full url pointing to a linux x64 distribution of docker machine. Once set docker_machine_version will be ingored. For example the GitLab version, https://gitlab-docker-machine-downloads.s3.amazonaws.com/v0.16.2-gitlab.2/docker-machine. |
string |
"https://gitlab-docker-machine-downloads.s3.amazonaws.com/v0.16.2-gitlab.2/docker-machine" |
no |
docker_machine_egress_rules | List of egress rules for the docker-machine instance(s). | list(object({ |
[ |
no |
docker_machine_iam_policy_arns | List of policy ARNs to be added to the instance profile of the docker machine runners. | list(string) |
[] |
no |
docker_machine_instance_type | Instance type used for the instances hosting docker-machine. | string |
"m5.large" |
no |
docker_machine_options | List of additional options for the docker machine config. Each element of this list must be a key=value pair. E.g. '["amazonec2-zone=a"]' | list(string) |
[] |
no |
docker_machine_role_json | Docker machine runner instance override policy, expected to be in JSON format. | string |
"" |
no |
docker_machine_security_group_description | A description for the docker-machine security group | string |
"A security group containing docker-machine instances" |
no |
docker_machine_spot_price_bid | Spot price bid. The maximum price willing to pay. By default the price is limited by the current on demand price for the instance type chosen. | string |
"on-demand-price" |
no |
docker_machine_version | By default docker_machine_download_url is used to set the docker machine version. Version of docker-machine. The version will be ingored once docker_machine_download_url is set. |
string |
"" |
no |
enable_asg_recreation | Enable automatic redeployment of the Runner ASG when the Launch Configs change. | bool |
true |
no |
enable_cloudwatch_logging | Boolean used to enable or disable the CloudWatch logging. | bool |
true |
no |
enable_docker_machine_ssm_access | Add IAM policies to the docker-machine instances to connect via the Session Manager. | bool |
false |
no |
enable_eip | Enable the assignment of an EIP to the gitlab runner instance | bool |
false |
no |
enable_forced_updates | DEPRECATED! and is replaced by enable_asg_recreation. Setting this variable to true will do the opposite as expected. For backward compatibility the variable will remain some releases. Old desription: Enable automatic redeployment of the Runner ASG when the Launch Configs change. |
string |
null |
no |
enable_kms | Let the module manage a KMS key, logs will be encrypted via KMS. Be-aware of the costs of an custom key. | bool |
false |
no |
enable_manage_gitlab_token | Boolean to enable the management of the GitLab token in SSM. If true the token will be stored in SSM, which means the SSM property is a terraform managed resource. If false the Gitlab token will be stored in the SSM by the user-data script during creation of the the instance. However the SSM parameter is not managed by terraform and will remain in SSM after a terraform destroy . |
bool |
true |
no |
enable_ping | Allow ICMP Ping to the ec2 instances. | bool |
false |
no |
enable_runner_ssm_access | Add IAM policies to the runner agent instance to connect via the Session Manager. | bool |
false |
no |
enable_runner_user_data_trace_log | Enable bash xtrace for the user data script that creates the EC2 instance for the runner agent. Be aware this could log sensitive data such as you GitLab runner token. | bool |
false |
no |
enable_schedule | Flag used to enable/disable auto scaling group schedule for the runner instance. | bool |
false |
no |
environment | A name that identifies the environment, used as prefix and for tagging. | string |
n/a | yes |
extra_security_group_ids_runner_agent | Optional IDs of extra security groups to apply to the runner agent. This will not apply to the runners spun up when using the docker+machine executor, which is the default. | list(string) |
[] |
no |
gitlab_runner_egress_rules | List of egress rules for the gitlab runner instance. | list(object({ |
[ |
no |
gitlab_runner_registration_config | Configuration used to register the runner. See the README for an example, or reference the examples in the examples directory of this repo. | map(string) |
{ |
no |
gitlab_runner_security_group_description | A description for the gitlab-runner security group | string |
"A security group containing gitlab-runner agent instances" |
no |
gitlab_runner_security_group_ids | A list of security group ids that are allowed to access the gitlab runner agent | list(string) |
[] |
no |
gitlab_runner_version | Version of the GitLab runner. | string |
"14.0.1" |
no |
instance_role_json | Default runner instance override policy, expected to be in JSON format. | string |
"" |
no |
instance_type | Instance type used for the GitLab runner. | string |
"t3.micro" |
no |
kms_alias_name | Alias added to the kms_key (if created and not provided by kms_key_id) | string |
"" |
no |
kms_deletion_window_in_days | Key rotation window, set to 0 for no rotation. Only used when enable_kms is set to true . |
number |
7 |
no |
kms_key_id | KMS key id to encrypted the CloudWatch logs. Ensure CloudWatch has access to the provided KMS key. | string |
"" |
no |
log_group_name | Option to override the default name (environment ) of the log group, requires enable_cloudwatch_logging = true . |
string |
null |
no |
metrics_autoscaling | A list of metrics to collect. The allowed values are GroupDesiredCapacity, GroupInServiceCapacity, GroupPendingCapacity, GroupMinSize, GroupMaxSize, GroupInServiceInstances, GroupPendingInstances, GroupStandbyInstances, GroupStandbyCapacity, GroupTerminatingCapacity, GroupTerminatingInstances, GroupTotalCapacity, GroupTotalInstances. | list(string) |
null |
no |
overrides | This maps provides the possibility to override some defaults. The following attributes are supported: name_sg overwrite the Name tag for all security groups created by this module. name_runner_agent_instance override the Name tag for the ec2 instance defined in the auto launch configuration. name_docker_machine_runners ovverrid the Name tag spot instances created by the runner agent. |
map(string) |
{ |
no |
permissions_boundary | Name of permissions boundary policy to attach to AWS IAM roles | string |
"" |
no |
role_tags | Map of tags that will be added to the role created. Useful for tag based authorization. | map(string) |
{} |
no |
runner_agent_uses_private_address | Restrict the runner agent to the use of a private IP address. If runner_agent_uses_private_address is set to false it will override the runners_use_private_address for the agent. |
bool |
true |
no |
runner_ami_filter | List of maps used to create the AMI filter for the Gitlab runner docker-machine AMI. | map(list(string)) |
{ |
no |
runner_ami_owners | The list of owners used to select the AMI of Gitlab runner docker-machine instances. | list(string) |
[ |
no |
runner_iam_policy_arns | List of policy ARNs to be added to the instance profile of the gitlab runner agent ec2 instance. | list(string) |
[] |
no |
runner_instance_ebs_optimized | Enable the GitLab runner instance to be EBS-optimized. | bool |
true |
no |
runner_instance_enable_monitoring | Enable the GitLab runner instance to have detailed monitoring. | bool |
true |
no |
runner_instance_metadata_options_http_endpoint | Enable the Gitlab runner agent instance metadata service. The allowed values are enabled, disabled. | string |
"enabled" |
no |
runner_instance_metadata_options_http_tokens | Set if Gitlab runner agent instance metadata service session tokens are required. The allowed values are optional, required. | string |
"optional" |
no |
runner_instance_spot_price | By setting a spot price bid price the runner agent will be created via a spot request. Be aware that spot instances can be stopped by AWS. Choose "on-demand-price" to pay up to the current on demand price for the instance type chosen. | string |
null |
no |
runner_root_block_device | The EC2 instance root block device configuration. Takes the following keys: device_name , delete_on_termination , volume_type , volume_size , encrypted , iops , throughput , kms_key_id |
map(string) |
{} |
no |
runner_tags | Map of tags that will be added to runner EC2 instances. | map(string) |
{} |
no |
runners_additional_volumes | Additional volumes that will be used in the runner config.toml, e.g Docker socket | list(any) |
[] |
no |
runners_concurrent | Concurrent value for the runners, will be used in the runner config.toml. | number |
10 |
no |
runners_disable_cache | Runners will not use local cache, will be used in the runner config.toml | bool |
false |
no |
runners_add_dind_volumes | Add certificates and docker.sock to the volumes to support docker-in-docker (dind) | bool |
false |
no |
runners_docker_runtime | docker runtime for runners, will be used in the runner config.toml | string |
"" |
no |
runners_ebs_optimized | Enable runners to be EBS-optimized. | bool |
true |
no |
runners_environment_vars | Environment variables during build execution, e.g. KEY=Value, see runner-public example. Will be used in the runner config.toml | list(string) |
[] |
no |
runners_executor | The executor to use. Currently supports docker+machine or docker . |
string |
"docker+machine" |
no |
runners_gitlab_url | URL of the GitLab instance to connect to. | string |
n/a | yes |
runners_helper_image | Overrides the default helper image used to clone repos and upload artifacts, will be used in the runner config.toml | string |
"" |
no |
runners_iam_instance_profile_name | IAM instance profile name of the runners, will be used in the runner config.toml | string |
"" |
no |
runners_idle_count | Idle count of the runners, will be used in the runner config.toml. | number |
0 |
no |
runners_idle_time | Idle time of the runners, will be used in the runner config.toml. | number |
600 |
no |
runners_image | Image to run builds, will be used in the runner config.toml | string |
"docker:18.03.1-ce" |
no |
runners_install_amazon_ecr_credential_helper | Install amazon-ecr-credential-helper inside userdata_pre_install script |
bool |
false |
no |
runners_limit | Limit for the runners, will be used in the runner config.toml. | number |
0 |
no |
runners_machine_autoscaling | Set autoscaling parameters based on periods, see https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersmachine-section | list(object({ |
[] |
no |
runners_max_builds | Max builds for each runner after which it will be removed, will be used in the runner config.toml. By default set to 0, no maxBuilds will be set in the configuration. | number |
0 |
no |
runners_monitoring | Enable detailed cloudwatch monitoring for spot instances. | bool |
false |
no |
runners_name | Name of the runner, will be used in the runner config.toml. | string |
n/a | yes |
runners_output_limit | Sets the maximum build log size in kilobytes, by default set to 4096 (4MB) | number |
4096 |
no |
runners_post_build_script | Commands to be executed on the Runner just after executing the build, but before executing after_script. | string |
"\"\"" |
no |
runners_pre_build_script | Script to execute in the pipeline just before the build, will be used in the runner config.toml | string |
"\"\"" |
no |
runners_pre_clone_script | Commands to be executed on the Runner before cloning the Git repository. this can be used to adjust the Git client configuration first, for example. | string |
"\"\"" |
no |
runners_privileged | Runners will run in privileged mode, will be used in the runner config.toml | bool |
true |
no |
runners_pull_policy | pull_policy for the runners, will be used in the runner config.toml | string |
"always" |
no |
runners_request_concurrency | Limit number of concurrent requests for new jobs from GitLab (default 1) | number |
1 |
no |
runners_request_spot_instance | Whether or not to request spot instances via docker-machine | bool |
true |
no |
runners_root_size | Runner instance root size in GB. | number |
16 |
no |
runners_services_volumes_tmpfs | n/a | list(object({ |
[] |
no |
runners_shm_size | shm_size for the runners, will be used in the runner config.toml | number |
0 |
no |
runners_token | Token for the runner, will be used in the runner config.toml. | string |
"__REPLACED_BY_USER_DATA__" |
no |
runners_use_private_address | Restrict runners to the use of a private IP address. If runner_agent_uses_private_address is set to true (default), runners_use_private_address will also apply for the agent. |
bool |
true |
no |
runners_volumes_tmpfs | n/a | list(object({ |
[] |
no |
schedule_config | Map containing the configuration of the ASG scale-in and scale-up for the runner instance. Will only be used if enable_schedule is set to true. | map(any) |
{ |
no |
secure_parameter_store_runner_sentry_dsn | The Sentry DSN name used to store the Sentry DSN in Secure Parameter Store | string |
"sentry-dsn" |
no |
secure_parameter_store_runner_token_key | The key name used store the Gitlab runner token in Secure Parameter Store | string |
"runner-token" |
no |
sentry_dsn | Sentry DSN of the project for the runner to use (uses legacy DSN format) | string |
"__SENTRY_DSN_REPLACED_BY_USER_DATA__" |
no |
subnet_id_runners | List of subnets used for hosting the gitlab-runners. | string |
n/a | yes |
subnet_ids_gitlab_runner | Subnet used for hosting the GitLab runner. | list(string) |
n/a | yes |
tags | Map of tags that will be added to created resources. By default resources will be tagged with name and environment. | map(string) |
{} |
no |
userdata_post_install | User-data script snippet to insert after GitLab runner install | string |
"" |
no |
userdata_pre_install | User-data script snippet to insert before GitLab runner install | string |
"" |
no |
vpc_id | The target VPC for the docker-machine and runner instances. | string |
n/a | yes |
Name | Description |
---|---|
runner_agent_role_arn | ARN of the role used for the ec2 instance for the GitLab runner agent. |
runner_agent_role_name | Name of the role used for the ec2 instance for the GitLab runner agent. |
runner_agent_sg_id | ID of the security group attached to the GitLab runner agent. |
runner_as_group_name | Name of the autoscaling group for the gitlab-runner instance |
runner_cache_bucket_arn | ARN of the S3 for the build cache. |
runner_cache_bucket_name | Name of the S3 for the build cache. |
runner_eip | EIP of the Gitlab Runner |
runner_role_arn | ARN of the role used for the docker machine runners. |
runner_role_name | Name of the role used for the docker machine runners. |
runner_sg_id | ID of the security group attached to the docker machine runners. |
This project exists thanks to all the people who contribute.
Made with contributors-img.