timebertt/ci-infra

Test infrastructure for the Gardener project.

ci-infra

This repository contains configuration files for the testing and automation needs of the Gardener project.

⚠️ Warning 🚧

This is currently under construction / in evaluation phase.

CI Job Management

Gardener uses a prow instance at [prow.gardener.cloud] to handle CI and automation for parts of the project. Everyone can participate in a self-service PR-based workflow, where changes are automatically deployed after they have been reviewed. All job configs are located in [config/jobs].

How to setup

1. Create the prow cluster and prow workload cluster.

   $ kubectl apply -f config/clusters/shoot.yaml
   $ kubectl apply -f config/clusters/shoot-workload.yaml

2. Create the prow namespace in the prow cluster:

   $ kubectl apply -f config/prow/cluster/prow_namespace.yaml

3. Create the test-pods namespace in the workload/build cluster:

   $ kubectl apply -f config/prow/cluster/build

4. Create the required secrets (mainly in the prow cluster):

• gardener-prow-storage (Service account with Storage Admin permissions for GCS bucket, according to test-infra guide, needs to be present in the prow namespace and in the test-pods namespace in both clusters)
• github-app (according to test-infra guide)
• github-oauth-config (according to test-infra guide)
• hmac-token

  $ kubectl -n prow create secret generic hmac-token --from-literal=hmac=$(openssl rand -hex 20)

• oauth-cookie-secret

  $ kubectl -n prow create secret generic oauth-cookie-secret --from-literal=secret=$(openssl rand -base64 32)

• kubeconfig (ref test-infra guide, needs to be present in the prow and test-pods namespace of the prow cluster)
  • add two contexts: the prow cluster as gardener-prow-trusted and the build/workload cluster as gardener-prow-build
  • gardener-prow-trusted context should use the in-cluster ServiceAccount token and CA file, so that all Prow components are bound to their respective RBAC roles
  • gardener-prow-build needs to be bound to the cluster-admin role. The gencred utility can be used to easily create a ServiceAccount and ClusterRoleBinding and retrieve the ServiceAccount token.
  • Template:

    apiVersion: v1
    kind: Config
    current-context: gardener-prow-build # default cluster
    contexts:
    - name: gardener-prow-trusted
      context:
        cluster: gardener-prow-trusted
        user: gardener-prow-trusted-token
    - name: gardener-prow-build
      context:
        cluster: gardener-prow-build
        user: gardener-prow-build-token
    clusters:
    - name: gardener-prow-trusted
      cluster: # in-cluster config
        server: 'https://kubernetes.default.svc'
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    - name: gardener-prow-build
      cluster:
        server: <<workload-cluster-api-server-address>>
        certificate-authority-data: <<base64-encoded-CA-bundle>>
    users:
    - name: gardener-prow-trusted-token
      user:
        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token # use in-cluster config
    - name: gardener-prow-build-token
      user:
        token: <<service-account-token-with-cluster-admin-permissions>> # generated via gencred

1. Deploy Prow components. The initial deployment has to be done manually, later on changes to the components will be automatically deployed once merged into master.

   $ ./config/prow/deploy.sh

2. Bootstrap Prow configuration/jobs. This initial configuration has to be done manually, later on changes to configuration and jobs will be automatically applied by the updateconfig plugin once merged into master.

   $ ./hack/boostrap-config.sh