hanefi/the-process

Trust it.

Table of Contents

• Introduction
• 1. Images
  • 1.1 How circleci uses our images in testing
• 2. How to update images
  • 2.1 How to update a postgres minor version in images
  • 2.2 How to add a major postgres major
• 3. How to publish a change

Introduction

This repository contains the source code for docker images that are used in citus testing. The images are pushed to docker hub. There is no hooking logic between this repository and the docker hub account. It is used purely for the storage of our images source code.

1. Images

The extbuilder image is the core image that other jobs depend on in our tests. The extbuilder:

While building the image:

• Installs postgres majors' pg_config. We only need pg_config to generate citus artifacts.

While running the container:

• Installs and tars the checked out citus version for the postgres versions. This is done so that the other jobs can install citus easily by untarring without the need to do make install. We need to do this step in the running container, not while building the image, so that the built citus version is the checked out citus code.
• Creates build-{pg_version} folders with citus configured, so these folders have the necessary Makefile.global to run the tests. These folders are generated for each postgres version. For example, at the time of writing there are 3 folders with build-10, build-11 and build-12.

The general process for other images are similar, which is:

While building the image:

• Install postgres major which is taken as an argument PG_MAJOR.

While running the container:

• Untar the citus tar to install citus for the postgres version that the image will use. For example, if postgres 11 is used, the tar is named install-11.tar.
• The script (install-and-test-ext) takes the target name as an argument, and the following command is used to run the test:

gosu circleci make -C "${CIRCLE_WORKING_DIRECTORY}/build-${PG_MAJOR}/src/test/regress" "${@}"

• Finally regression.diffs is printed if it exists, which indicates that there was a problem during the tests.

Because of the specific logic that we have to run tests faster, we have a docker image debbuilder that generates a debian package for postgresql-server-dev-{pg-major}. The generated package is in the built docker image, specifically in {container-name}/home/circleci/debs. The package is pushed to the-process. This package place is added to apt source list and pinned with with the highest priority so that it is chosen over the standard postgresql-server-dev-{pg-major} package.

The installation of postgres packages is at the time of building the image to reduce testing time. However this means that once the image is built the postgres version is fixed. You will need to rebuild the image if you want to upgrade the image, upgrading the image for a minor version is easier compared to upgrading a major version.

1.1. How circleci uses our images in testing

Our dockerhub account has all the images here except the debbuilder. When a change is committed, circleci:

• Pulls the https://hub.docker.com/r/citus/extbuilder image, generates the citus artifacts which other images will use for testing.
• All of our test jobs wait for extbuilder to be completed. Once that is done, all of the jobs are run in parallel without any specific order(It is possible that some jobs are waiting because of parallelism limit in our circleci).
• All the test jobs use the artifacts that are generated with extbuilder.

Refer to images for more details.

2. How to update images

While updating images, if the image uses python such as failtester, you will need to update the requirements.txt as well. We are using pipenv for development in citus but we generate requirements.txt file with pipenv to install our python dependencies. Running something like the following should generate requirements.txt:

 pipenv lock --requirements > requirements.txt

Make sure that you include the commit id of citus at the beginning of the generated requirements.txt. If you are adding a new python dependency you will need to repeat this process for images to be updated and have the correct dependencies.

2.1 How to update postgres minor version in images

To update a postgres minor version, you will need to generate a debian package:

• Generate postgresql-server-dev-{pg-major} package with debbuilder image.
• The generated package will be in {container-name}/home/circleci/debs. One way of getting the package from the docker image is:
  • Run the container with docker run -it {tag-name} bash
  • Find the name of your container with docker ps
  • Copy the debs folder from docker container to your local docker cp {container-name}:/home/circleci/debs .
• Upload the package to the-process. Make sure that you choose debian stretch while uploading. In order to upload you can:
  • Upload the package file from the UI by clicking to Upload image
  • Or you can use the package cloud cli.

After adding the package to package cloud:

And for all images follow the instructions on how to publish a change

2.2. How to add a postgres major

If you want to add support for a major postgres major, like pg-13 you will need to do the steps that are not limited to:

• Generate postgresql-server-dev-{pg-major} package with debbuilder image. Make sure to add the package index for pg-major in /etc/apt/sources.list.d/pgdg.list. You can see find the related part here.

• The generated package will be in {container-name}/home/circleci/debs. One way of getting the package from the docker image is:

  • Run the container with docker run -it {tag-name} bash
  • Find the name of your container with docker ps
  • Copy the debs folder from docker container to your local docker cp {container-name}:/home/circleci/debs .

• Upload the package to the-process. Make sure that you choose debian stretch while uploading. In order to upload you can:

  • Upload the package file from the UI by clicking to Upload image
  • Or you can use the package cloud cli.

• Add pg-major to extbuilder in its script so that citus artifacts are generated for that pg_major too.

• Add new pg-major package index to sources.list:

# add pgdg repo to sources
echo "Writing /etc/apt/sources.list.d/pgdg.list..." >&2
echo "deb http://apt.postgresql.org/pub/repos/apt/ ${codename}-pgdg main ${PG-MAJOR}" > \
    /etc/apt/sources.list.d/postgresql.list

• Build exttester with pg-major:

cd exttester
docker build --tag=citus/exttester-{pg-major} . --build-arg PG_MAJOR={pg-major}

• Build failtester with pg-major:

cd failtester
docker build --tag=citus/failtester-{pg-major} . --build-arg PG_MAJOR={pg-major}

3. How to publish a change

If you want to make a change and push it to docker hub, you should do:

• Clone the repository and make the necessary changes
• Build the image

docker build --tag={repositoryName}/{imageName} .

• Login to your account

docker login # enter your crendentials

• Push the image

docker push {repositoryName}/{imageName}