/retorch

RETORCH: Resource-aware End-to-End Test Orchestration

Primary LanguageJavaApache License 2.0Apache-2.0

Status Quality Gate Status Maven Central (annotations)

RETORCH: Resource-aware End-to-End Test Orchestration

This repository contains a series of components of the RETORCH End-to-End (E2E) test orchestration framework. It's primary goal is to optimize E2E test execution by reducing both the execution time and the number of unnecessary Resource1 redeployment's.

NOTE: The repository is a work in progress, the initial version only made available the annotations, and currently we're migrating the orchestration generator module. Additional components will be added in future releases.

Contents

Quick-start

  • Add the dependency io.github.giis-uniovi:retorch-annotations to the pom.xml of your SUT.
  • Add the annotations to the test classes as indicated below
  • Configure the E2E test suite as indicated below
  • Execute the orchestration generator and generate the pipelining-scripting code
  • [TO-DO]

RETORCH Annotations

The RETORCH framework provides a set of custom annotations to define and manage Resources used in end-to-end testing. These annotations allow testers to group, schedule, and characterize Resources. To execute test cases using RETORCH, each test case must be annotated with at least one access mode and resource.

The tester needs to specify the access mode using the following attributes:

  • resID: Resource identifier for the access mode.
  • concurrency: The upper bound of test cases that can access the resource concurrently.
  • sharing: Allows sharing the resource between multiple test cases.
  • accessMode: The type of access mode performed by the test case.
@AccessMode(resID = "LoginService", concurrency = 10, sharing = true, accessMode = "READONLY")

Each access mode annotation corresponds to a specific resource, which must be annotated with the following attributes:

  • resID: A unique identifier for the resource.
  • replaceable: A list of Resources that can replace the current one.
@Resource(resID = "LoginService", replaceable = {})

The following code snippets illustrate a test case annotated with multiple Resources and access modes:

@Resource(resID = "LoginService", replaceable = {})
@AccessMode(resID = "LoginService", concurrency = 10, sharing = true, accessMode = "READONLY")
@Resource(resID = "OpenVidu", replaceable = {"OpenViduMock"})
@AccessMode(resID = "OpenVidu", concurrency = 10, sharing = true, accessMode = "NOACCESS")
@Resource(resID = "Course", replaceable = {"Forum"})
@AccessMode(resID = "Course", concurrency = 10, sharing = true, accessMode = "READONLY")
@ParameterizedTest
@MethodSource("data")
void forumLoadEntriesTest(String usermail,String password,String role){
        this.user=setupBrowser("chrome",TJOB_NAME+"_"+TEST_NAME,usermail,WAIT_SECONDS);
        driver=user.getDriver();
        this.slowLogin(user,usermail,password);
        }

RETORCH Orchestration

The RETORCH framework provides a generator that creates the Execution Plan, along with the required pipelining and script files for execution in a CI environment. The generation of scripts and pipelining code is based on the Access Modes annotated within the test cases and the Resource information specified in [SUT_NAME]SystemResources.json.

The RETORCH orchestration generator requires 4 inputs:

  • The annotated E2E test cases with the RETORCH access modes into a single module Maven project.
  • A file with the Resources in JSON format.
  • A properties file with the Environment configuration.
  • A custom docker-compose.yml file.

Given these inputs, the generator gives as output the necessary scripting code and the Jenkinsfile to execute the E2E test suite into a Continuous Integration system.

Prepare the E2E Test suite

The first step is to create several folders to store the configurations and place the docker-compose.yml in the single module project root. The resulting directory tree might look like as:

.
├── src
├── docker-compose.yml
├── retorchfiles/
│   ├── configurations/
│   └── customscriptscode/
  • The retorchfiles/ directory would contain all the configuration files and scripting snippets that would be used to generate the pipelining code and the scripts to set up, deploy, and tear down the different Resources and TJob. Contains two subdirectories:
    • configurations/: stores the Resource and CI configuration files.
    • customscriptscode/: stores the different script snippets for the tear down, set up and environment.
  • The docker-compose.yml in the root of the project.
  • The different project directories and files.

The following subsections explain how to create each configuration file and how to prepare the docker-compose.yml file.

Create the Resource.json file

The Resource file must be placed in the retorchfiles/configurations/ and named with the system or test suite name, followed by SystemResources.json. This file contains a map with a series of Resources, using their unique ResourceID as a key. For each Resource the tester needs to specify the following attributes:

  • resourceID: A unique identifier for the Resource.
  • replaceable: A list of Resources that can replace the current one.
  • hierarchyParent: A resourceID of the hierarchical parent of the Resource.
  • elasticityModel: The elasticity model of the Resource, is composed by the following attributes:
    • elasticityID: A unique identifier for the elasticity model.
    • elasticity: Integer with the available Resources.
    • elasticityCost: Instantiation cost of each Resource.
  • resourceType: String with the type of the Resource(e.g. LOGICAL, PHYSICAL or COMPUTATIONAL).
  • minimalCapacities: List with the Minimal Capacities required by the Resource; each Capacity is composed by:
    • name: String between "memory", "processor" and "storage".
    • quantity: float with the amount of Capacity Required.
  • dockerImage: String with the concatenation of the placeholder name in the docker-compose, "[IMG:]" and the image name.

The following snippet shows an example of two Resources declared in the JSON file:

{ 
  "userservice": {
  "hierarchyParent": ["mysql"],
  "replaceable": [],
  "elasticityModel": {"elasticityID": "elasmodeluserservice", "elasticity": 5, "elasticityCost": 30.0},
  "resourceType": "LOGICAL", "resourceID": "userservice",
  "minimalCapacities": [
    {"name": "memory", "quantity": 0.2929}, 
    {"name": "processor", "quantity": 0.2}, 
    {"name": "storage", "quantity": 0.5}],
    "dockerImage": "userservice[IMG:]wigo4it/identityserver4:latest"
},
  "frontend": {
    "hierarchyParent": [], "replaceable": [],
    "elasticityModel": {"elasticityID": "elasmodelfrontend", "elasticity": 1, "elasticityCost": 300.0},
    "resourceType": "LOGICAL", "resourceID": "frontend",
    "minimalCapacities": [
      {"name": "memory", "quantity": 2}, 
      {"name": "processor", "quantity": 1}, 
      {"name": "storage", "quantity": 0.88}],
    "dockerImage": "frontend[IMG:]nginx:latest"
  }
  
}

Create the retorchCI.properties file

The CI file must be placed in retorchfiles/configurations/, namely retorchCI.properties containing several parameters related to the SUT and the Continuous Integration Infrastructure, these parameters are the following:

  • agentCIName: the specific Jenkins agent used to execute the test suite.
  • sut-wait-html: state in the frontend (HTML displayed) when the SUT is ready to execute the test SUITE.
  • sut-location: location of the docker-compose.yml file used to deploy the SUT.
  • docker-frontend-name: ID of the container used as frontend .
  • docker-frontend-port: PORT on which the frontend container is available.
  • external-binded-port: EXTERNAL PORT where the frontend is made available (if its available).
  • external-frontend-url: EXTERNAL URI where the frontend is made available.
  • testsBasePath: Path to the Java project root.

The following snippet provides an example of how this file looks like:

  agentCIName=any
  sut-wait-html=<title>Hello World</title>
  sut-location=$WORKSPACE
  docker-frontend-name=https://sutexample-
  docker-frontend-port=5000
  external-binded-port=
  external-frontend-url=
  testsBasePath=./

Preparing the docker-compose.yml file

The orchestration generator also requires to parametrize the docker-compose.yml used to deploy the application by means including the necessary environment variables in the containers names and URIs, as well as the placeholders of the images specified above. Examples of the necessary changes in the docker-compose.yml can consulted in the FullTeaching and eShopOnContainers repositories:

(Optional) Specify script snippets to include in the set-up tear-down and environment

The RETORCH orchestration generator allows to specify scripting code/commands to be included in the generated set up, tear down, and the environment declaration of each TJob. To include it, the tester must create the following files in retorch\customscriptscode:

  • custom-tjob-setup: Contains the custom set up code (e.g. declare some environment variable specific for each TJob) or custom logging systems.
  • custom-tjob-teardown: Contains the custom tear down code (e.g. save some generated outputs).
  • custom.env: Contains configurations and environment variables common to all TJobs.

Examples of the three snippets files can be consulted in FullTeaching Test Suite and eShopOnContainers.

Once created the different properties and configuration files, the single module directory tree might look like:

.
├── src
├── docker-compose.yml
├── retorchfiles/
│   ├── configurations/
│   │   ├── [SUTNAME]SystemResource.json
│   │   └── retorchCI.properties
│   └── customscriptscode/
        ├── custom-tjob-setup
        ├── custom-tjob-teardown
        └── custom.env

Executing the Orchestration generator

Once all the files created and the docker-compose.yml is prepared, to execute the generator we only need to create a main class and instantiate an OrchestratinGenericToolbox object. Calling the generateJenkinsFile() method with the following parameters:

  • packageRoute: String that specifies the complete package name to the E2E annotated tests folder.
  • systemName: String that specifies the system name, must correspond with the name used in the Resources JSON file.
  • jenkinsFilePath: String with the location where the Jenkinsfile will be created (usually the root of the project: ./).

The following code snippet shows an example of the method invocation:

import giis.retorch.orchestration.main.OrchestrationGenericToolBox;

public class SutExampleRetorchMain {

   public static void main(String[] args)  {
       OrchestrationGenericToolBox toolBox = new OrchestrationGenericToolBox();
       toolBox.generateJenkinsfile("giis.sutexample.e2e.functional.tests", "sutexample", "./");
   }
}

RETORCH Orchestration generator outputs

The generator provides four different outputs: the pipelining code, the necessary scripts to set up, tear down and execute the TJobs(/retorchfiles/tjoblifecycles), the infrastructure(/retorchfiles/coilifecycles) and the different environment files of each TJob (/retorchfiles/envfiles) :

  • Jenkinsfile: located in the root of the project, contains the pipelining code with the different stages in sequential-parallel that perform the different TJob lifecycle stages.
  • /retorchfiles/tjoblifecycles and /retorchfiles/coilifecycles contains the set up, execution, and tear down scripts for the TJobs and infrastructure
  • /retorchfiles/envfiles: contains the generated custom environment of each TJob.

Contributing

See the general contribution policies and guidelines for giis-uniovi at CONTRIBUTING.md.

Contact

Cristian Augusto - augustocristian@uniovi.es - Software Engineering Research Group (GIIS) - University of Oviedo, ES

Citing this work

RETORCH E2E Test Orchestration framework:

RETORCH*: A Cost and Resource aware Model for E2E Testing in the Cloud:

Acknowledgments

This work has been developed under the TestBUS (PID2019-105455GB-C32) and project supported by the Ministry of Science and Innovation (SPAIN)

Footnotes

  1. Henceforth, we will use the term "Resources" (capitalized) when referring to the ones required by the E2E test suite. ↩