Testing framework and E2E tests are written in Go using Ginkgo and Gomega frameworks to cover Red Hat AppStudio. It is recommended to install AppStudio in E2E mode, but the E2E suite can be also usable in development and preview modes.
- Instrumented tests with Ginkgo 2.0 framework. You can find more information in Ginkgo documentation.
- Uses client-go to connect to OpenShift Cluster.
- Ability to run the E2E tests everywhere: locally(CRC/OpenShift local), OpenShift Cluster, OSD...
- Writes tests results in JUnit XML/JSON file to a custom directory by using
--ginkgo.junit(or json)-report
flag. - Ability to run the test suites separately.
Whnen you want to run the E2E tests for AppStudio you need to have installed tools(in Requirements chapter), installed the AppStudio in E2E mode and compiled e2e-appstudio
binary.
Requirements for installing AppStudio in E2E mode and running the E2E tests:
- An OpenShift 4.9 or higher Environment (If you are using CRC/OpenShift Local please also review optional-codeready-containers-post-bootstrap-configuration)
- A machine from which to run the install (usually your laptop) with required tools:
- A properly setup Go workspace using Go 1.17 is required
- The OpenShift Command Line Tool (oc)
- yq
- jq
- git
- Tokens
- Github Token with the following permissions
repo
delete_repo
- Valid quay token where to push AppStudio components images generated by the e2e framework
- Github Token with the following permissions
Before executing the e2e suites you need to have deployed AppStudio in E2E Mode to your cluster.
-
Before deploying AppStudio in E2E mode you need to login to your OpenShift cluster with OpenShift Command Line Tool as
admin
(by defaultkubeadmin
):oc login -u <user> -p <password> --server=<oc_api_url>
-
Setup environment variables for GitHub and Quay.io tokens(tokens are required, you can also set more variables, see the table below):
`export GITHUB_TOKEN=ghp_Iq...` `export QUAY_TOKEN=ewogI3...`
The following environments are used to launch the Red Hat AppStudio installation in E2E mode and the tests execution(tokens are also used for running the tests):
Variable | Required | Explanation | Default Value |
---|---|---|---|
GITHUB_TOKEN |
yes | A github token used to create AppStudio applications in github | '' |
QUAY_TOKEN |
yes | A quay token to push components images to quay.io. Note the quay token must be your dockerconfigjson encoded in base64 format, e.g. ewogI3dJhdXRocyI6I... |
'' |
GITHUB_E2E_ORGANIZATION |
no | GitHub Organization where to create/push Red Hat AppStudio Applications | redhat-appstudio-qe |
QUAY_E2E_ORGANIZATION |
no | Quay organization where to push components containers | redhat-appstudio-qe |
E2E_APPLICATIONS_NAMESPACE |
no | Name of the namespace used for running HAS E2E tests | appstudio-e2e-test |
PRIVATE_DEVFILE_SAMPLE |
no | The name of the private git repository used in HAS E2E tests. Your GITHUB_TOKEN should be able to read from it. | https://github.com/redhat-appstudio-qe/private-quarkus-devfile-sample |
QUAY_OAUTH_USER |
no | A valid quay robot account username to make quay oauth | '' |
QUAY_OAUTH_TOKEN |
no | A valid quay quay robot account token to make oauth against quay.io. | '' |
- Install dependencies:
# Install dependencies
$ go mod tidy
# or go mod tidy -compat=1.17
# Copy the dependencies to vendor folder
$ go mod vendor
-
Install Red Hat AppStudio in e2e mode with install script in scripts folder. The e2e framework will use by default the
redhat-appstudio-qe
GitHub organization(you can change the GitHub organization by environment variableGITHUB_E2E_ORGANIZATION
).make local/cluster/prepare
or
./scripts/install-appstudio-e2e-mode.sh install
More information about how to deploy Red Hat AppStudio are in the infra-deployments repository.
You can use scripts to build and run the tests:
make local/test/e2e
Or build and run the tests without scripts:
-
Install dependencies and build the tests:
# Install dependencies $ go mod tidy # Copy the dependencies to vendor folder $ go mod vendor # Create `e2e-appstudio` binary in bin folder. Please add the binary to the path or just execute `./bin/e2e-appstudio` $ make build
-
Run the e2e tests: The
e2e-appstudio
command is the root command that executes all test functionality. To obtain all available flags for the binary please use--help
flags. All ginkgo flags and go tests are available ine2e-appstudio
binary.`./bin/e2e-appstudio`
The instructions for every test suite can be found in the tests folder, e.g. has Readme.md. You can also specify hich tests you want to run using labels or Ginkgo Focus.
Load tests for AppStudio are also in this repository. More information about load tests are in LoadTests.md.
Overview for OpenShift CI and AppStudio E2E tests is in OpenshiftCI.md. How to install E2E binary is in Installation.md.
- Create test folder under tests folder:
tests/[<application-name>]...
, e.g. has.tests/has
- all tests used owned by AppStudio application service team
- Every test package should be imported to
cmd/e2e_test.go
, e.g. has. - Every new test should have correct labels.
- Every test should have meaningful description with JIRA/GitHub issue key.
- (Recommended) Use JIRA integration for linking issues and commits (just add JIRA issue key in the commit message). You can found more information about GitHub-JIRA integration here.
// cmd/e2e_test.go
package common
import (
// ensure these packages are scanned by ginkgo for e2e tests
_ "github.com/redhat-appstudio/e2e-tests/tests/common"
_ "github.com/redhat-appstudio/e2e-tests/tests/has"
)
For reporting issues with e2e tests please use RHDP JIRA project - please use labels appstudio
and quality
. If the issues is also relevant to CI - for example it is blocking PRs, the also use label appstudio-e2e-tests-known-issues
.
There is launch configuration in .vscode/launch.json
called Launch demo suites
.
Running this configuration, you'll be asked for github token and then e2e-demos suite will run with default configuration.
If you want to run/debug different suite, change -ginkgo.focus
parameter in .vscode/launch.json
.