EAP ADDITIONAL TESTSUITE

THE JBOSS TESTSUITE TO DEVELOP TESTS AGAINST INFINITE NUMBER OF JBOSS SERVERS

A PROJECT UNDER THE ΙΔΕΑ STATEMENT

THIS REPO IS DIFFERED AS FAR AS DEV IS CONCERNED...

PLEASE REFER TO : https://github.com/EAT-JBCOMMUNITY/EAT, https://github.com/panossot/EAT-1, https://github.com/panossot/EAT

ALL THE COMMITS ON THIS REPO SHOULD HAVE AN EQUIVALENT COMMIT AT ONE OF THE REPOS MENTIONED ABOVE ...

An additional JBOSS testsuite in order to facilitate Qualtity Testing.

Write your tests once and run them against any version of EAP and WILDFLY application server.

In order to synchronize the dependency versions of eap-additional-testsuite with the component versions of the equivalent server you can run the additional-testsuite using -Dstandalone or -Dserver-integration property.

Standalone Mode

If you enable standalone mode (-Dstandalone), please make sure that the server parent pom is available locally or remotely as an artifact.

Server-Integration Mode

If you enable server-integration mode (-Dserver-integration), please make sure that the server BOM is available locally or remotely as an artifact.

Testing EAP

Make sure that JBOSS_FOLDER environment variable is set with the path to your JBOSS EAP directory.
Make sure that JBOSS_VERSION environment variable is set with the version of JBOSS EAP Server.
Build and run the additional testsuite activating the EAP profile (-Deap -Dstandalone).
If you want to use http add -Dmaven.repository.protocol=http (Section : Using HTTPS below)
If you would like/need to disable snapshot versions add -DDISABLE_SNAPSHOT_VERSIONS=true

Testing Wildfly

Make sure that JBOSS_FOLDER environment variable is set with the path to your WILDFLY directory.
Make sure that JBOSS_VERSION environment variable is set with the version of WILDFLY Server.
Build and run the additional testsuite activating the WILDFLY profile (-Dwildfly -Dstandalone).
If you want to use http add -Dmaven.repository.protocol=http (Section : Using HTTPS below)
If you would like/need to disable snapshot versions add -DDISABLE_SNAPSHOT_VERSIONS=true

Testing EAP or Wildfly with specific JDK version

Make sure that JBOSS_FOLDER is set with the path to your EAP OR WILDFLY directory.
Make sure that JBOSS_VERSION environment variable is set with the version of JBOSS EAP OR WILDFLY Server.
Make sure that JAVA_HOME is pointing to the jdk of desired version.
Build and run the additional testsuite activating the EAP or WILDFLY specific jdk version profile (-Deap-jdk8, -Dwildfly-jdk8).
If you want to use http add -Dmaven.repository.protocol=http (Section : Using HTTPS below)
If you would like/need to disable snapshot versions add -DDISABLE_SNAPSHOT_VERSIONS=true

Testing with Open Liberty

export OPENLIBERTY_VERSION='the version of OpenLiberty features-bom you would like to use'
mvn clean install -Dopenliberty (It should be executed with appropriate apache maven version, e.g. 3.5.3 works)

Using HTTPS

After changing from HTTP to HTTPS, the maven repositories for Eap may not host a valid public SSL certificate, which will lead to maven failed to build the project. There are 3 options to fix that if it happens:

Specify the -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true in the maven build command line to bypass the SSL certificates check (This is suitable for local workstation case)
Specify -Dmaven.repository.protocol=http to switch back to HTTP
Import the server certificate into your local trusted store.

ADVANTAGES

Writing the tests once and testing against infinite number of Application Servers.
Having all the tests at one place.
Comparison of the servers based on the testsuite.
Guarding against regression.
Faster convergence among the servers.
Comparison of the servers based on tests of the past and the present.
Addition of tests with possible future features that are not at the moment available.
It makes possible to push a testcase of a fix regarding a specific component of the server, without the component version to have been updated at the server pom.
Ability to merge tests from remote testsuites.

MOTIVATION

If a test is developed for server X then it can be automatically tested against all the other servers.

HOW TO DEBUG (for testable deployments and non-marked as @RunAsClient tests)

Start the server (that you have defined in JBOSS_FOLDER) with the --debug 'port' option.
Connect the Debugger to the Remote VM inside the server using some IDE.
Activate the equivalent debug profile at your IDE (eg eap7.testsuite).
Set your breakpoints.
And then start the debugging executing the specific test which is errorous.

HOW TO DEBUG (for non-testable deployments and marked as @RunAsClient tests)

Activate the equivalent debug profile at your IDE (eg eap7.testsuite).
Set your breakpoints.
Run your tests from command line adding -Dmaven.surefire.debug="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8787 -Xnoagent -Djava.compiler=NONE" property. e.g. mvn -Dmaven.surefire.debug="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8787 -Xnoagent -Djava.compiler=NONE" test -Dwildfly -Dstandalone
Connect the Debugger to the Remote VM using some IDE.
Debug the specific test which is errorous.

HOW TO BUILD EAP ADDITIONAL TESTSUITE INSIDE AN IDE (ECLIPSE, IntelliJ IDEA, NETBEANS)

Before starting the IDE, go to your IDE directory and export JBOSS_FOLDER and JBOSS_VERSION environment variables at file /etc/environment and execute the command "source /etc/environment".
Start your IDE from command line and load the project.
Go to Modules directory and activate the desired server profile (eg wildfly.testsuite). Then activate the standalone profile at the server dir that has appeared.
Make sure that the server parent pom is available locally or remotely as an artifact.
Modify the run/build configurations in order to use the the desired goals, profiles and environment variables
Build your project.

HOW TO PROCESS THE SOURCES INSIDE AN IDE

In order to process the eap-additional-testsuite sources, please, after following the steps of the section "HOW TO BUILD EAP ADDITIONAL TESTSUITE INSIDE AN IDE (ECLIPSE, IntelliJ IDEA, NETBEANS)" (activating the equivalent server profile and the standalone profile, etc), proceed to the following actions depending on the way of processing and the IDE that you choose :

1st WAY OF PROCESSING (This way applies for all the IDEs)

Load the subcategory level where exist the tests that you would like to process.
Process your sources.
Do not forget to copy the modified sources at the Modules Level (modules/src/main/java). In a different case, if you run the project from the Parent Level, your modified sources will be overriden by the ones that exist at the Modules Level (Risk of losing your modifications).

2nd WAY OF PROCESSING

Please, go to the Server Level (e.g. Wildfly Server Level) and follow the actions depending on the IDE:

Eclipse :
- Go to the Server Level Project -> Properties -> Java Build Path. Under the Source Tab click the Link Source Button. Linked folder location should be "PATH_TO_PROJECT/eap-additional-testsuite/modules/src/main/java" and "Update exclusion filters in other source folders to solve nesting" should be chosen. Apply the changes. The sources should be, now, visible in your IDE to process.
- The .project file should look like :

<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
	<name>test</name>
	<comment></comment>
	<projects>
	</projects>
	<buildSpec>
		<buildCommand>
			<name>org.eclipse.m2e.core.maven2Builder</name>
			<arguments>
			</arguments>
		</buildCommand>
	</buildSpec>
	<natures>
		<nature>org.eclipse.jdt.core.javanature</nature>
		<nature>org.eclipse.m2e.core.maven2Nature</nature>
	</natures>
	<linkedResources>
		<link>
			<name>java</name>
			<type>2</type>
			<location>"PATH TO THE LINKED SOURCE FILES"</location>
		</link>
	</linkedResources>
</projectDescription>

Add the JRE System Library in your project.
Add the following lines in the .classpath file :

<classpathentry exported="true" kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER"/>
<classpathentry exported="true" kind="con" path="org.eclipse.m2e.MAVEN2_CLASSPATH_CONTAINER">
		<attributes>
			<attribute name="maven.pomderived" value="true"/>
		</attributes>
</classpathentry>

IntelliJ IDEA : Go to the Server Level Project -> Project Structure -> Sources, remove the current context root and add "PATH_TO_PROJECT/eap-additional-testsuite/modules/src/main/java" as the context root. Apply the changes. The sources should be, now, visible in your IDE to process under "java" directory.
Netbeans : The sources should be visible in your IDE to process.

Important note :

If the sources are to be modified or extended, the working directory of the run/build configuration should be set to the eap-additional-testsuite parent directory, in order that any changes in the sources would be propagated to the specific server sources.
The sources to be processed should be the ones linked to modules/src/main/java directory.

EAP-ADDITIONAL-TESTSUITE INTEGRATED INSIDE THE SERVERS (HOW TO USE) - AVAILABLE SOON

Send your PR containing the test to eap-additional-testsuite.
Before sending the server PR inform the server that you would like to include a specific PR of eap-additional-testsuite, adding the following lines in git-sub-modules/git-sub-module.sh :
- git submodule update --init --remote
- cd ../testsuite/addtional-testsuite/eap-additional-testsuite/
- git fetch origin refs/pull/8/head && git checkout FETCH_HEAD (declare that you would like to include PR 8 of eap-additional-testsuite)
Commit the changes on the server and send the server PR.
The specific PR of the test will be tested along with the server PR.

EXAMPLE OF USAGE

Supposing that we would like to add Mtom TestCase for Wildfly (master) and Eap 7 (7.x) Servers in Eap Additional Testsuite, we would follow the steps bellow :

Add the source code of the test in modules/src/main directory (both for Wildfly and Eap7 - write your code once) and annotate all the classes used in the test with the @EapAdditionalTestsuite annotation, specifing the locations of the Servers that the classes will be distributed to, as a String Array attribute (e.g. @EapAdditionalTestsuite Example).
Add the resources in Wildfy Mtom recource directory (for Wildfly) and Eap7 Mtom recource directory (for Eap7)
Add any additional dependencies in Wildfy pom (for Wildfly) and Eap7 pom (for Eap7)
Enable the Mtom TestCase in Wildfy Web Configuration pom (for Wildfly) and Eap7 Web Configuration pom (for Eap7)
Build the Eap Additional Testsuite for Wildfy and Eap7 (having set the equivalent JBOSS_FOLDER and JBOSS_VERSION environment variables) :
- mvn clean install -Dwildfly -Dstandalone (for Wildfly)
- mvn clean install -Deap7 -Dstandalone (for Eap7)

OTHER FEATURES

Infinite number of servers could be added in Eap-Additional-Testsuite, while the tests included could be tested against all these servers.
Eap-Additional-Testsuite could be a very helpful tool in combination with the documentation. In parallel with the documentation, examples of use cases could be provided, which could be tested against all the servers.
Eap-Additional-Testsuite is the first component to change the relation of Component -> Server to Component <-> Server, as this component can be considered as a part of a Server, but also a Server can be considered as a part of Eap-Additional-Testsuite.
The AT Structures can be used with any Software Program language.

I NEED ASSISTANCE TO ADD A NEW TEST

If any assistance is needed to add a new test in Eap Additional Testsuite, please add it to the following directories, denoting the versions of the servers within the tests should be tested (eg EAP 7.1.1.DR1, etc) :

ADD A SINGLE VERSION TEST WHICH WILL BE TESTED FOR MULTIPLE VERSIONS

For people not familiar with EAT, they can add a single version test (as usual) here : https://github.com/jboss-set/eap-additional-testsuite/tree/master/SingleVersionTestInclusion , commenting where this should be included (servers, server-versions, etc). e.g.

/**

@author author_name
@EATSERVERS : Wildfly,Eap72x,Eap72x-Proposed
@EATSERVERMODULE : basic
@EATLOWERLIMITVERSIONS : 18.0.0, 7.2.3, 7.2.3
@EATUPPERLIMITVERSIONS : 18.0.0, 7.2.4, 7.2.4

Then this test can be added to the right places to be distributed for the specific server versions.

HOW TO MERGE TESTS FROM REMOTE TESTSUITES

Set the GITHUB_REPO environment variable with the url of the remote testsuite repo (e.g. export GITHUB_REPO=https://github.com/panossot/RemoteTestRepo.git).
Set the SUBDIR environment variable with the subdirectory path of the repo you would like to copy to modules/src/main/java direcory of EAT (e.g. export SUBDIR=RemoteTestRepo/src/java/main/org).
Run EAT as usual.

INNOVATION

The innovative part of EAT is creating the test once and testing with any version of the tested software. It may be firstly applied for the JBOSS Servers, but, in general, a similar structure, can be used for creating tests about any software with multiple versions or for multiple software programs that have a part of the testsuite in common.

Another innovative part of the AT structure is that using the annotation @ATFeature, it can create dynamic subsets of tests in order to test dynamically created software programs (using provisioning tools) according to a specific list of features / characteristics.

FUTURE DEVELOPMENT

Use the generalized additional testsuite structure in order to include Fedora and Openshift tests for multiple versions of Fedora and Openshift.
Create different categories of ATs (Additional Testsuites) : e.g. ATs for features could be different from ATs for bugs.
Use AT structures for testing containers (e.g. using Arquillian cube)
Create a framework that will combine all ATs (Additional Testsuites), e.g. EAT, JBTAT, etc. The user would be able to add a test to whichever testsuite he/she would choose. Furthermore, the user should be able to add more ATs in this framework, according to the software project of usage.
The past sources could contain only the differences.
Distribution of tests based on if a specific commit is merged.
The servers could build only the modules related to the proposed modification changes.
Make it more user friendly (for whoever is not familiar with its usage), using a pre-entry step, by adding a testcase, as usual, to a single-version server pre-testsuite. Then the test case will be added in the Eap-Additional-Testsuite sources for multiversion usage.
Create a subcategory for a collection of components.
Show how it can test multi-versioned software with multiple components that are dynamically updated in real time using component chains.
Use arquillian cube for container testing.
Make dependencies fully dynamic
Create scripts that wrap the multiversion testsuite with the IDEs.
NEXT GENERATION AT STRUCTURES : The tests/methods (that may use features/dependencies from a provisioning tool/dependency-feature analyzer) should be independent modules that will have dependencies (metadata could be used) on libraries and other methods (which may also use some features/dependencies from the provisioning tool/dependency-feature analyzer and which could have their own dependencies). The tool should compose the appropriate test cases and distribute them to the appropriate subcategories in accordance with the available features/dependencies. The dependencies could also be defined automatically when using a dependency-feature analyzer. This way the annotations for enabling the tests could be automated (and become non mandatory in some cases).

MOJO DOCUMENTATION

Eap Additional Testsuite Mojo Document

EAT with PROTEAN/QUARKUS DOCUMENTATION

Eap Additional Testsuite with Protean/Quarkus

EAT with OPENSHIFT

EAT with Openshift

EAP-ADDITIONAL-TESTSUITE PRESENTATION

Eap Additional Testsuite Presentation

EAT JAM

EAP-ADDITIONAL-TESTSUITE VIDEO PRESENTATION

Eap Additional Testsuite Video

Eap Additional Testsuite Quick Presentation

EAP-ADDITIONAL-TESTSUITE WORKSHOP

Eap Additional Testsuite Workshop

Eap Additional Testsuite Workshop (Download)

EAP-ADDITIONAL-TESTSUITE as a COLLABORATION PROJECT

Many thanks to Carlo de Wolf for his ideas and help and for including EAT under the JBoss-SET umbrella. Also, many thanks to the University of Peloponnese (Department of Informatics and Telecommunications) for the theory of AT Structures.

License

GNU Lesser General Public License Version 2.1 (GPL Cooperation Commitment)

Note

02-10-2019 : Clarification - Commits of user root (with invalid e-mail) were sent by user panossot .

RanabirChakraborty/eap-additional-testsuite