This project provides Technology Compatibility Kit (TCK) for Google App Engine (GAE) API.
The goals of the project are
- comprehensive GAE API test coverage
- GAE API usage examples
- common GAE API test environment
Project can either be used by external vendors (not Google) implementing the GAE API - such as CapeDwarf and AppScale, or developers writing frameworks on top of GAE API, or simply using GAE API.
e.g. the easiest way to file a bug against GAE API is to provide a valid failing test
Visit our web site at http://www.appengine-tck.org for more info.
The content of this repository is released under the Apache License 2.0 as provided in the LICENSE file that accompanied this code.
By submitting a "pull request" or otherwise contributing to this repository, you agree to license your contribution under the license mentioned above.
This software uses a number of other works, the license terms of which are documented in the NOTICE file that accompanied this code.
common
- common test code; base tests and multisuite supportcore
- core test code; black list, endpoints, SQL, modules, ...env
- custom environment hooks; GAE SDK, Appspot, CapeDwarf, AppScale, ...ext
- external (useful) tests; e.g. DataNucleus, MapReduce, Objectify, ...site
- reporting site, exposing data from TCK TeamCity CI runstests
- the main TCK API testsutils
- helper utils; code coverage plugin, bytecode transforemers
Java JDK7+ and Maven 3.x+. And of course the environment you want to test. It is assumed you have working knowledge of Maven and Git.
To simply run the tests against the sdk you can follow the instructions below. If you are submitting a test or making any other contribution see How to Submit Tests
git clone https://github.com/GoogleCloudPlatform/appengine-tck.git
cd appengine-tck
mvn clean install
cd tests
mvn clean install -Psdk,multisuite -Dappengine.sdk.root=<PATH_TO_SDK>
A summary of each API (SUCCESS/FAILURE) will be listed at the end, and the raw results will be located in tests/appengine-tck-[test-package]/target/surefire-reports.
To generate an html report:
mvn surefire-report:report
There will be a report for each api located in tests/appengine-tck-[test-package]/target/site/surefire-report.html.The details of the failures will are at the bottom.
The rest of this document goes into the details of running and writing tests.
The whole project is fully Mavenized, so for the simple build the process is obvious:
cd appengine-tck
mvn clean install
This builds the default profiles, which includes compiling the tests, but not running them. And we also output the API code coverage metrics; see console output. Before running any tests you need to do this step in order to build the framework. Then you can go into specific test directories and run only those tests. If you don't do this step and go straight to the test directory you will get compilation errors.
All tests are build using Arquillian and ShrinkWrap. Each different environment has a custom Arquillian container implementation.
GAE container implementations can be found here: https://github.com/alesj/arquillian-container-gae
CapeDwarf uses plain JBossAS Arquillian container implementations - remote and managed. Where we recommend remote for manual (simple) testing, and managed for automated (e.g. TeamCity, Hudson, ...) testing.
For each set of tests we want to get code coverage, needs a coverage.txt file, where we list all the classes / interfaces whose usage we track. Deprecated classes, interfaces, and methods will not be reported.
e.g. TaskQueue code covereage
file: appengine-tck/tests/appengine-tck-memcache/coverage.txt
com.google.appengine.api.taskqueue.Queue
This will print out all Queue interface usage in our TaskQueue tests.
You can either see the results in console while the build is running, or at the end open index.html file in TCK root. A csv file is also generated which can be imported into a spreadsheet. Each test directory has its own index.html and csv reports.
You can override which file name is used to lookup to list classes / interfaces. This is done either by changing the coverage plugin's configuration in pom.xml or using -Dcoverage.file system property. A copy of the coverage file is located in the respective test directories.
cd appengine-tck
mvn clean install -Dcoverage.file=coverage.txt.all
Or just:
cd appengine-tck
mvn clean install
The report is located here: appengine-tck/index.html
We can also explicitly exclude some of the API methods with exclusions.txt file (deprecated methods are already excluded by default).
e.g. excluding com.google.appengine.api.urlfetch.URLFetchServiceFactory's constructor
com.google.appengine.api.urlfetch.URLFetchServiceFactory@<init>@()V
The default exclusion information must follow this pattern:
<fqn class name>@<method name>@<descriptor>
Running the tests is same as building the project, but you also need to specify the environment you want to test.
First, as always, build at the root directory:
cd appengine-tck
mvn clean install
Then either go to the tests directory to run all the tests or the directory of the specific api, e.g. tests/appengine-tck-memcache
cd appengine-tck/tests or cd appengine-tck/tests/appengine-tck-memcache
This are the current enviroments:
-
GAE SDK
mvn clean install -Psdk -Dappengine.sdk.root=<PATH_TO_SDK>
-
GAE Appspot
mvn clean install -Pappspot -Dappengine.sdk.root=<PATH_TO_SDK> -Dappengine.userId=<USER_ID> -Dappengine.password=<USER_PASSWORD> -Dappengine.appId=<APPSPOT_APP_ID>
-
JBoss CapeDwarf (remote)
mvn clean install -Pcapedwarf
Where you need a running instance of CapeDwarf environment to be present.
-
JBoss CapeDwarf (managed)
mvn clean install -Dcapedwarf.home
Where you need CapeDwarf distribution to be present under "capedwarf.home" directory.
We learned how to run the tests. But that will deploy one test class per deployment, which can be time consuming for environments like Appspot.
In order to handle all tests within one deployment, we added multisuite
profile.
mvn clean install -Psdk,multisuite -Dappengine.sdk.root=<PATH_TO_SDK>
This will deploy all tests (per API) in a single .war file, and then test them as usual.
Note: as we create an uber .war from all tests, not all tests can be included. Tests that are not capable of running inside uber .war, must be marked with @IgnoreMultisuite.
Each set of tests that we want to run -Pmultisuite on them, requires a multisuite.marker file in tests' root directory. In this marker we can override test class regexp patter or scanning strategy; see code for more details.
In case we have some failures in some API tests, but we still want to run the whole TCK, you can ignore failures with this Maven flag
-Dmaven.test.failure.ignore=true
Since it's all Mavenized, this is trivial, as Surefire plugin already supports this.
cd appengine-tck/tests/appengine-tck-[package]
mvn clean install -Psdk -Dappengine.sdk.root=<PATH_TO_SDK> -Dtest=<TEST_SIMPLE_NAME>
To run a specific test method.
mvn clean install -Psdk -Dappengine.sdk.root=<PATH_TO_SDK> -Dtest=<TEST_SIMPLE_NAME>#testTheMethod
By default all tests are checked for @WithinNamespace annotation, which instructs tests to be wrapped with different namespace setup; the array of namespaces from @WithinNamespace' value is used for this setup.
If you want to ignore this namespace testing, use -Dtck.ignore.within.namespace=true flag when running the tests.
mvn clean install -Dtck.ignore.within.namespace=true -Pcapedwarf
During the tests we can fire TestLifecycle events. Each enviroment can add (via ServiceLoader pattern) multiple TestLifecycle implementations; e.g. one per particular event type.
e.g.
public class CapeDwarfTestContextEnhancer implements TestLifecycle {
public void before(TestLifecycleEvent event) {
if (event instanceof TestContextLifecycleEvent) {
TestContextLifecycleEvent tcle = (TestContextLifecycleEvent) event;
enhance(tcle.getTestContext());
}
}
Current API tests:
- AppIdentity
- Blobstore
- Channel
- Datastore
- Images
- Logging
- Memcache
- Modules
- OAuth
- Prospective Search
- Search
- TaskQueue
- UrlFetch
- Users
- XMPP
This module is testing GAE built-in core functionality beyond APIs.
Current core tests:
- Endpoints support
- Miscellaneous; e.g. black list usage, etc
- SQL
All these tests also run by default, same way as API tests. (we used to require explicit profile, this is not the case anymore)
mvn clean install -Pcapedwarf
Simply run tests against selected environment / profile; e.g. CapeDwarf in this case.
Here we try to gather any useful tests that will help make GAE API implementation environments better.
Current external tests:
- GAE DataNucleus Plugin
- GAE MapReduce Library
- Google Cloud Storage Client
- JUnit Example
MapReduce and Google Cloud Storage tests run by default, where DataNucleus and Example are under custom profiles.
e.g. in the case of DataNucleus GAE plugin we use -Pdatanucleus
mvn clean install -Pcapedwarf,datanucleus
This will run all DataNucleus tests against CapeDwarf environment.
e.g. running MapReduce tests against SDK
mvn clean install -Psdk -Dappengine.sdk.root=<PATH_TO_SDK>
Note: adapting the tests to run against real environment can sometime be a huge (bytecode) hack. :-)
More info can be found here: How to run external tests
In case you already have existing JUnit tests, running against local test environment; e.g. similar to DataNucleus plugin, you can - with some bytecode magic - easily re-use those tests to run against any environment, in-container this time.
To see how that's done, see JUnit example or GAE DataNucleus Plugin external testing.
See appengine-tck/tests/appengine-tck-example for a bare bones illustration of how to write a new test package.
As we already mentioned, all tests are Arquillian and ShrinkWrap based, with JUnit being the actual test framework used, and Maven Surefire plugin to run it all.
But there are a few guidelines we need to follow, specially with regard to "multisuite".
Few basic Arquillian rules:
- each test class needs to have @RunWith(Arquillian.class) in its class hierarchy
- there must be exactly one (in whole class hierarchy) static method marked with @Deployment, returning WebArchive instance
Few basic ShrinkWrap rules:
- do not forget to add base test classes to deployment, only actual test class is added by default
- make sure .war's META-INF is in WEB-INF/classes/META-INF, as per web spec
Few basic "multisuite" rules:
- we merge all WebArchive instances from all tests into one single uber WebArchive instance
- luckily there is a notification filter, which spits out warning by default if resources overlap
- notifications can be changed to failure if needed, for more strict resource usage in tests
- test classes can be excluded from "multisuite" with @IgnoreMultisuite
Why all this fuss about "multisuite"? Imagine two different web.xml files, where one test needs servlet A, and the other one needs servlet B. In multisuite, there is only one web.xml - first come, first serve - so one test could miss its servlet. Which means we need to carefully craft test recources. This mostly results in more middle / abstract classes, holding common resources.
To make things a bit easier, we already introduced a few abstract / helper classes and methods.
- each test class should extend TestBase class
- to get WebArchive instance use TestBase::getTckDeployment with proper TestContext instance
- if any custom behavior needs to be added per test environment, add this via TestLifecycles::before or TestLifecycles::after
- LibUtils class can be used to grab whole libraries / jars from Maven and attach them to .war as library (WEB-INF/lib)
- one can add its own multisuite ScanStrategy and NotificationFilter; see ScanMultiProvider class for more details
The maven-checkstyle-plugin is enabled for this project, and will report a build failure if there is a violation.
Running it is the same as code coverage:
cd appengine-tck
mvn clean install
To generate an html report:
mvn site
The results will be located in tests/appengine-tck-[test-package]/target/site/checkstyle.html.
To summarize, there are 3 kind of reports that are generated:
- Code Coverage,
- Checkstyle,
- Test Results.
cd appengine-tck
mvn clean install
Code coverage will be located in index.html file.
To generate Checkstyle reports:
mvn site
See tests/appengine-tck-[test-package]/target/site/checkstyle.html
cd appengine-tck/tests
mvn clean install -Psdk,... (run tests)
mvn surefire-report:report
See tests/appengine-tck-[test-package]/target/site/surefire-report.html
Site module is made up from two parts:
- reporting web app
- a TeamCity plugin that pushes the build info to the reporting web app when done.
The reporting web app shows nice pie charts for all environments, with drill-down info details on failed tests.