Overview

For an overview about the ROS build farm including how to deploy the necessary machine see the ROS buildfarm wiki page.

This repository implementation for deploying servers for the ROS buildfarm. It typically requires the configurations given as an example in buildfarm_deployment_config.

After the servers have been provisioned you will then want to see the ros_buildfarm project for how to configure Jenkins with ROS jobs.

If you are going to use any of the provided infrastructure please consider signing up for the build farm mailing list (https://groups.google.com/d/forum/ros-sig-buildfarm) in order to receive notifications e.g. about any upcoming changes.

Process

To effectively use this there will be three main steps.

Provision the hardware/VM instances.
Fork this repository and update the configuration.
Deploy the forked configuration onto the machines.

At the end of this process you will have a Jenkins master, a package repository, and N jenkins-slaves.

Provisioning

The following EC2 instance types are recommended when deploying to Amazon EC2. They are intended as a guideline for choosing the appropriate parameters when deploying to other platforms.

Master

Memory	30Gb
Disk space	200Gb
Recommendation	r3.xlarge

Slave

Disk space	200Gb+
Recommendation	c3.large or faster

Repo

Disk space	100Gb
Recommendation	t2.medium

Forking (or not)

The config repository will contain your secrets such as private keys and access tokens, keep it private! You should make a copy of the config repository and make it private. Unfortunately you can't use the "Fork" button on GitHub and then make it private.

To create a private fork.

Create a new empty private repo
Push from a clone of the public repo into the private repo.

Access

To give access to your private repo you will need to authenticate.

The below example has setup the config repo with token access. And embedded the token in the below URLs. Keep this token secret!

Updating values

It is recommended to change all the security parameters from this configuration. In particular you should change the following:

On all three:

jenkins::slave::ui_pass
This is the passworkd for the slave to access the master
user::admin::password_hash
On the master this should be the hashed password from above
The easiest way to create this is to setup a jenkins instance. Change the password, then copy the string out of config file on the jenkins server.
autoreconfigure::branch
If you are forking into a repo and using a different branch name, update the autoreconfigure command to point to the right branch.
ssh_keys
Configure as many public ssh-keys as you want for administrators to log in. It's recommended at least one for root.
On the repo machine make sure there is at least one key for the jenkins-slave user matching the ssh private key jenkins::private_key provisioned on the master.

ssh_keys:
    'admin@foobar':
        ensure: present
        key: AAAAB3NzaC1yc2EAAAADAQABAAABAQC2NOaRsdZqqTrCwNR77AQIqwAPYkDfiL1Ou7Pi/qaW9S7UU0Y1KAQ6kWhgJc9RtOhbZKGHbFTqSLT4235TkmPvlZbV2bK8ZViBmqQ3r8vDMhC/+p9Ec9SP8sjv6JcIEWOy5zXPnB3OnHHWXmvZP47rjJY0l76F71fZt3vlvyjz7IrikULmuKcyrE+zulmbSTtfGZhxQRPxZDO/RiOemCPsYo5u/rUMjWH+CkEI0swQlM6QIvjWdfYtNwQT9yo53MXFy5jodhW4YOOncKE4RMOI9Lmu6jE0GmdmSEv486R4ot6iWanx2hk/46zlmX1kSKGWObRdH57H/xIAxvw+PiTd
        type: ssh-rsa
        user: root
    'upload_access@buildfarm':
        ensure: present
        key: AAAAB3NzaC1yc2EAAAADAQABAAABAQC2NOaRsdZqqTrCwNR77AQIqwAPYkDfiL1Ou7Pi/qaW9S7UU0Y1KAQ6kWhgJc9RtOhbZKGHbFTqSLT4235TkmPvlZbV2bK8ZViBmqQ3r8vDMhC/+p9Ec9SP8sjv6JcIEWOy5zXPnB3OnHHWXmvZP47rjJY0l76F71fZt3vlvyjz7IrikULmuKcyrE+zulmbSTtfGZhxQRPxZDO/RiOemCPsYo5u/rUMjWH+CkEI0swQlM6QIvjWdfYtNwQT9yo53MXFy5jodhW4YOOncKE4RMOI9Lmu6jE0GmdmSEv486R4ot6iWanx2hk/46zlmX1kSKGWObRdH57H/xIAxvw+PiTd
        type: ssh-rsa
        user: jenkins-slave

On repo:

jenkins-slave::gpg_public_key
The GPG public key matching the private key. This will be made available for download from the repo for verification.
jenkins-slave::gpg_private_key
The GPG key with which to sign the repository.
master::ip
The IP address of the master instance.
jenkins-slave::reprepro_config
Fill in the correct rules for upstream imports. It should be a hash/dict item with the filename as the key, ensure, and content as elements like below. You can have as many elements as you want for different files.

jenkins-slave::reprepro_config:
    '/home/jenkins-slave/reprepro_config/empy_saucy.yaml':
        ensure: 'present'
        content: |
            name: empy_saucy
            method: http://packages.osrfoundation.org/gazebo/ubuntu
            suites: [saucy]
            component: main
            architectures: [i386, amd64, source]
            filter_formula: Package (% python3-empy)

On the master:

jenkins::private_ssh_key
The ssh private key will be provisioned as an ssh-credential available via the ssh-agent inside a jenkins jobs. This is necessary for access to push content onto the repo machine. It can also be used to access other machines from within the job execution environment. This will require deploying the matching public key to the other machines appropriately. Note: This value should be kept secret!
master::ip
The IP address of the master instance.
repo::ip
The IP address of the repository instance.
credentials::jenkins-slave::username
The name of the credentials
credentials::jenkins-slave::id
A UUID for the credentials in thef format 1e7d4696-7fd4-4bc6-8c87-ebc7b6ce16e5
credentials::jenkins-slave::passphrase
The hashed passphrase for the key. The UI puts this has in if there's now passphrase 4lRsx/NwfEndwUlcWOOnYg==
If you would like to modify these values from the default it will likely be easiest to boot an instance. Change the credentials via the UI, then grab the values out of the config file.

On the slave:

master::ip
The IP address of the master instance.
repo::ip
The IP address of the repository instance.
jenkins::slave::num_executors
The number of executors to instantiate on each slave. From current testing you can do one per available core, as long as at least 2GB of memory are available for each executor.

Deployment

Once you have customized all the content of the config repo on each provisioned machine run the following sequence of commands.

Master deployment

sudo su
cd
apt-get update
apt-get install -y git

# Customize this URL for your fork
git clone https://8d25f41a3ed71b0b9fc571c8a35bcb47fb4f6489@github.com/YOUR_ORG/buildfarm_deployment_config.git
cd buildfarm_deployment_config
./install_prerequisites.bash
./reconfigure.bash master

repo deployment

sudo su
cd
apt-get update
apt-get install -y git

# Customize this URL for your fork
git clone https://8d25f41a3ed71b0b9fc571c8a35bcb47fb4f6489@github.com/YOUR_ORG/buildfarm_deployment_config.git
cd buildfarm_deployment_config
./install_prerequisites.bash
./reconfigure.bash repo

slave deployment

sudo su
cd
apt-get update
apt-get install -y git

# Customize this URL for your fork
git clone https://8d25f41a3ed71b0b9fc571c8a35bcb47fb4f6489@github.com/YOUR_ORG/buildfarm_deployment_config.git
cd buildfarm_deployment_config
./install_prerequisites.bash
./reconfigure.bash slave

After Deployment

Now that you have a running system you will need to add jobs for one or more rosdistros. See the ros_buildfarm repo for more info.

For information on development and testing

See doc/DevelopmentTesting.md

tappan-at-git/buildfarm_deployment