OpenShift Pipelines E2E tests

Validation of OpenShift Pipeline releases using Gauge

Prerequisites

Go
Gauge
Clone this repository
Need OpenShift cluster
Download latest OpenShift Client for your operating system

Installation instructions

Install Gauge to /usr/local/bin by running

curl -SsL https://downloads.gauge.org/stable | sh

or install Gauge to a <custom path> using

curl -SsL https://downloads.gauge.org/stable | sh -s -- --location-[custom path]

For other installation methods, refer to the Gauge documentation.

Install plugins

GO111MODULE=off gauge install go
GO111MODULE=off gauge install html-report
GO111MODULE=off gauge install screenshot
GO111MODULE=off gauge install xml-report

(optional)

GO111MODULE=off gauge install reportportal

Run a specification

Refer to the Gauge documentation for general information about how to run specifications (aka specs).

Run tests for OpenShift Pipelines

Majority of tests assume that they run on an OpenShift cluster with already installed OpenShift Pipelines operator (e.g. latest release of a nightly build).

Operator installation, upgrade and uninstallation

Operator installation tests have to run as admin user

CATALOG_SOURCE=custom-operators CHANNEL=latest gauge run --log-level=debug --verbose --tags install specs/olm.spec
CATALOG_SOURCE=custom-operators CHANNEL=latest gauge run --log-level=debug --verbose --tags upgrade specs/olm.spec
gauge run --log-level=debug --verbose --tags uninstall specs/olm.spec

Notes:

CATALOG_SOURCE - catalog source name, redhat-operators for released versions, custom-operators for nightly builds

CHANNEL - channel to which the installation test is supposed to subscribe, e.g. latest or pipelines-1.9

Most common test sub-suites

The following tests have to run as admin user

gauge run --log-level=debug --verbose --tags e2e specs/metrics
gauge run --log-level=debug --verbose --tags 'e2e & tls' specs/triggers/eventlistener.spec
gauge run --log-level=debug --verbose --tags e2e specs/clustertasks/clustertask.spec
gauge run --log-level=debug --verbose --tags 'e2e & linux/amd64' specs/clustertasks/clustertask-multiarch.spec
gauge run --log-level=debug --verbose --tags e2e specs/operator/rbac.spec
gauge run --log-level=debug --verbose --tags e2e specs/operator/auto-prune.spec
gauge run --log-level=debug --verbose --tags e2e specs/operator/addon.spec

The following tests can run as both admin and regular/non-admin user

gauge run --log-level=debug --verbose --tags e2e specs/pipelines
gauge run --log-level=debug --verbose --tags 'e2e & !tls' specs/triggers
gauge run --log-level=debug --verbose --tags disconnected-e2e specs/clustertasks/clustertask.spec
gauge run --log-level=debug --verbose --tags 'e2e & !skip_linux/amd64' specs/clustertasks/clustertask-s2i.spec

Authoring a new test specification

Create or update a spec file in specs directory using Markdown syntax.
If necessary, create steps in a new or appropriate existing Go file in steps directory.
If necessary, create test resources in YAML in testdata directory.
If necessary, implement new steps using Go in new or appropriate existing file in pkg directory.

Running tests in a container

CI system is running these tests inside a container using image quay.io/openshift-pipeline/ci built using a Dockerfile hosted in a non-public repository.

cd <path_with_content_of_this_repo>
podman run --rm -it -v $KUBECONFIG:/root/.kube/config:z -v .:/root/release-tests:z -w /root/release-tests quay.io/openshift-pipeline/ci /bin/bash
gauge run ...

jkandasa/openshift-pipelines-release-tests