Magellan is a tool for massively-scaling your automated test suite, with added reliability. Run large test suites across across many environments (multiple browsers or versions, or multiple native iOS or Android devices) at the same time, in parallel, with a friendly command-line workflow that is both local development and continuous-integration friendly. Magellan is compatible with mocha (wd.js, webdriver.io, appium) tests ( example Mocha/wd project ) and Nightwatch.js tests ( example Nightwatch project ), and includes SauceLabs support. Through Magellan's mocha support, you can scale regular node.js test suites too.
- Parallel Test Runs
- Worker allocation and management with failed test retry.
- Network port management and testing, with isolated ports for mocking servers, individual (per-worker) selenium servers.
- Configurable worker count.
- Testing and debugging workflows. Run many tests at once, one test at a time, filter by tags, groups, etc.
- Suite run control: Bail likely-failing suite runs early, or bail upon first failure.
- Run many different parallel local browsers (eg: Chrome, Firefox, etc) all at the same time.
- Run many different parallel remote (SauceLabs) browsers.
- Integration Support
- Status reporter API with events streamed from workers, with some included reporters.
- Slack reporting support.
- Admiral reporting support.
- Plays well with CI (Jenkins, etc).
- SauceLabs Remote Browser and Device Support:
- Optional Sauce Connect tunnel management (
--create_tunnels). - Create lists of browser tiers or browser testing groups with browser profiles (eg: tier1 browsers, tier2 browsers, mobile browsers, vintage IE versions, etc).
- Manage not just browsers, but also devices for native application testing (iOS and Android)
- Optional Sauce Connect tunnel management (
- Can talk to a locks service to control saucelabs virtual machine usage (beta).
mocha:wd( example Mocha/wd project )webdriver.io( example Mocha/webdriver.io project )
Nightwatch.js( example Nightwatch project )node.js(non-browser) test suites (example project coming soon).appium.js(example project coming soon).
Magellan can best be described as a runner-runner. If you use mocha or nightwatch to run your current tests, then you can use Magellan to in turn run mocha or nightwatch and scale up your test runs.
When running tests with mocha, Magellan simply stacks on top of your existing suite:
Note: This is the preferred solution for working with mocha. If you want to spend less time handling selenium desiredCapabilities objects and starting/stopping Selenium, Magellan works much better with rowdy, which handles these tasks for you (for examples of this in action, see our example Mocha/wd project and example Mocha/webdriver.io project). That setup looks like this:
Magellan can also run Nightwatch.js test suites (please see our example Nightwatch project ), in which case your stack looks like this:
Finally, Magellan can also scale regular ol' node.js tests (no browsers or devices) using Mocha:
Magellan is a command line tool.
Note: The following examples assume you have ./node_modules/.bin in your PATH. If you don't have this rule in PATH, or are unable to add it, you can also run any of the examples below like this:
$ ./node_modules/.bin/magellanBy default, magellan will try to run your the fastest way possible, in parallel, in the phantomjs browser.
To execute your tests, run:
$ magellanYou can also run parallel tests on a real local browser:
# launch several instances of Chrome at once and run tests in parallel
$ magellan --browser=chrome
# launch several instances of Firefox at once and run tests in parallel
$ magellan --browser=firefoxmagellan can run your test suite across multiple browsers with one command:
# Run tests locally in both PhantomJS and Chrome
$ magellan --browser=chrome,phantomjs
# Run tests locally in Chrome and Firefox
$ magellan --browser=chrome,firefoxIn Mocha-based tests, Magellan will find tags in it() strings in the form of a tagname preceded by an @. For example, to define a test that would match with --tags=commerce or --tags=smoke, we would write:
it("should submit the purchase @commerce @smoke", function (done) {
...
});In Nightwatch-based tests, Magellan supports the standard Nightwatch convention of tags. To define a test that Magellan can match with --tags=commerce or --tags=smoke, we would write:
module.exports = new Test({
tags: ["commerce", "smoke"],
"Load the order page": function (client) {
...
},
"Submit the purchase": function (client) {
...
}
});To filter by one or more tags, run magellan with the --tag or --tags option:
# Specify one tag:
$ magellan --tag=customer
# Specify multiple tags:
$ magellan --tags=customer,guestTo limit the your tests by a file path prefix, use the --group option:
$ magellan --group=tests/SmokeThe above filter will match tests/product/tests/Smoke*
Filter options can be combined together. For example, --tags and --group work together:
$ magellan --group=tests/Smoke --tags=customer,guestTo run one specific test, use the --test flag with a path to the test file:
$ magellan --test=path/to/my/test.jsYou can run your entire suite serially (i.e. one at a time) and get live console output with the --serial option:
$ magellan --serialNote that --serial can be combined with --tag, --group and --test filters, as well as different --browser options.
Test suites that have one or more failing tests can be terminated faster to conserve resource usage (i.e. in CI) or obtain results on failing tests faster (i.e. during a developer workflow).
To terminate a run early as soon as any test has failed, use --bail_fast:
$ magellan --bail_fastTo terminate a run early if 10% of total test runs have failed (and at least 10 test runs have occurred), use --bail_early.
$ magellan --bail_earlyThis option allows for a build to run for longer but still terminate if a significant portion of the suite is failing. Use this option if you want to avoid running a long test run, but want to be able to potentially spot trends in failing tests.
You can control how long Magellan waits for a test to execute before explicitly killing it by supplying a bail_time argument. For example, to set bail time to 60 seconds:
$ magellan --bail_early --bail_time=60000A bail option does not have to be used to set bail time. For example:
$ magellan --bail_time=60000The bail_time setting can also be written to Magellan configuration. See
To use the same arguments on every magellan run, create a file called magellan.json and use the same arguments as magellan takes, but in JSON format. For example:
{
"bail_fast": true,
"max_workers": 5
}To supply a path to a different magellan configuration file, use the --config argument:
$ magellan --config=./alternate_config.jsonMagellan supports custom reporters. A custom reporter is a module that inherits from Magellan's base reporter class and can listen to a number of events and streams for monitoring test activity.
To include a custom reporter, add it to magellan.json like this:
{
"bail_fast": true,
"max_workers": 5,
"reporters": [
"./path/to/my/reporter",
"my_reporter_module"
]
}In the example above, ./path/to/my/reporter refers to a module in your test suite directory tree, whereas my_reporter_module is a module you have included in package.json
var BaseReporter = require("testarmada-magellan").Reporter;
var util = require("util");
var Reporter = function () {
};
util.inherits(Reporter, BaseReporter);
Reporter.prototype.listenTo = function (testRun, source) {
// Stream stdout and stderr directly to stdout, assuming this source is
// a process that has those properties.
if (source.stdout) {
source.stdout.pipe(process.stdout);
}
if (source.stderr) {
source.stderr.pipe(process.stderr);
}
};
module.exports = Reporter;When magellan is preparing a test run, it will call your reporter's implementation of listenTo(). Arguments are as follows:
sourceis a node event emitter that optionally has propertiesstdoutandstderr.testRunis an object with information about the specific test run this source is associated with. It has the following properties:path: The filesystem path to the actual test file. Useful as an identifier for a test.buildId: The id of the whole suite run.tempAssetPath: The path to place various temporary assets (logs, screenshots, configuration files, etc) generated by or for a specific test run.
At the moment, there is one event available, worker-status, which indicates whether a Magellan worker has started executing a test or finished executing a test. The example below illustrates how to unpack this event:
source.on("message", function (msg) {
if (msg.type === "worker-status") {
if (msg.status === "started") {
console.log("Test started: " + msg.name);
} else if (msg.status === "finished") {
console.log("Test finished: " + msg.name);
console.log(" Pass/fail: " + msg.passed);
}
}
});Some custom reporters need to initialize themselves asynchronously before listenTo can be called. To do this, the a reporter class can define an initialize() method, and return a Q promise:
initialize: function () {
var self = this;
var deferred = Q.defer();
console.log("Initializing reporter..");
createJob(function(err, jobId) {
if (err) {
deferred.reject(err);
} else {
self.jobId = jobId;
deferred.resolve();
}
});
return deferred.promise;
},In the above example, a reporter needs to create a job entry and obtain the id of the job before it can send information about running tests. Magellan will wait until initialize() resolves this promise before starting any tests.
Magellan supports running tests through SauceLabs remote browsers. To do this, the following environment variables should be set:
# Set SauceLabs Credentials
export SAUCE_USERNAME='xxxxxxxxxx'
export SAUCE_ACCESS_KEY='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
# Set Secure Tunnel Settings
# SauceConnect version for download
export SAUCE_CONNECT_VERSION=4.3.10A Sauce Connect tunnel prefix must be set for tunnel management to work when using --create_tunnels (see more on this below).
# Tunnel id prefix, Example: "my_tunnel", "qa_tunnel", etc
export SAUCE_TUNNEL_ID="xxxxxxxxx"
Magellan can query the SauceLabs API for a list of available browsers (for web testing) and devices (for native app testing in iOS and Android), and present them as a list of friendly browser ids:
$ magellan --list_browsersTo use a given SauceLabs browser, specify it when using the --sauce option:
$ magellan --sauce --browser=chrome_42_Windows_2012_R2_DesktopTo use multiple SauceLabs browsers and environments at the same time, simply list multiple ids:
$ magellan --sauce --browsers=chrome_42_Windows_2012_R2_Desktop,safari_7_OS_X_10_9_Desktop
Note: If you are building reporting or CI tools and want to use the same SauceLabs API and browser naming support toolset, check out guacamole.
NOTE: By default, Magellan assumes that tests run on an open network visible to SauceLabs.
If your tests are running in a closed CI environment not visible to the Internet, a tunnel is required from SauceLabs to your test machine (when using --sauce mode). To activate tunnel creation, use --create_tunnels. Magellan will create a tunnel for the test run. If you want to distribute your work load across more than one tunnel, specify the --max_tunnels=N option, like so:
$ magellan --sauce --browser=chrome_42_Windows_2012_R2_Desktop --create_tunnels --max_tunnels=4 --max_workers=16In the above example, 4 tunnels will be distributed amongst 16 workers.
To specify a locks server, set the environment variable LOCKS_SERVER:
export LOCKS_SERVER=http://locks.server.example:4765/
or use the --locks_server option:
$ magellan --locks_server=http://locks.server.example:4765 --sauce --browser=chrome_42_Windows_2012_R2_Desktop --create_tunnels --max_tunnels=4 --max_workers=16
To ensure that the SauceLabs display being used has enough resolution to support a given browser window size, use the --resolution option:
Single Sauce browser:
$ magellan --sauce --browser=chrome_42_Windows_2012_R2_Desktop --resolution=1024x768Multiple Sauce browsers:
$ magellan --sauce --browsers=chrome_42_Windows_2012_R2_Desktop,safari_7_OS_X_10_9_Desktop --resolution=1024x768In this case, 1024x768 is selected for chrome_42_Windows_2012_R2_Desktop and safari_7_OS_X_10_9_Desktop. If this resolution isn't available in all Sauce browser environments specified, Magellan will return an error.
For Sauce devices that support it, orientation is also supported with the --orientation option:
$ magellan --sauce --browser=iphone_8_2_iOS_iPhone_Simulator --orientation=landscapeSometimes it's useful to specify a list of environments with differing resolutions or orientations. For this case, Magellan supports profiles stored in magellan.json:
{
"profiles": {
"my_profile_1": [
{ "browser": "chrome_42_Windows_2012_R2_Desktop", "resolution": "2560x1600" },
{ "browser": "chrome_42_Windows_2012_R2_Desktop", "resolution": "800x600" },
{ "browser": "ipad_8_2_iOS_iPad_Simulator", "orientation": "landscape" },
{ "browser": "ipad_8_2_iOS_iPad_Simulator", "orientation": "portrait" },
{ "browser": "iphone_8_2_iOS_iPhone_Simulator", "orientation": "portrait" }
],
"tier_2_browsers": [
{ "browser": "safari_7_OS_X_10_9_Desktop" },
{ "browser": "IE_8_Windows_2008_Desktop" },
{ "browser": "IE_9_Windows_2008_Desktop" }
]
}Notice that resolution and orientation are optional. Multiple definitions of the same browser running in different resolutions or orientations are permitted. To select these profiles, we call:
$ magellan --profile=tier_2_browsers
# or
$ magellan --profile=my_profile_1
# or specify multiple profiles
$ magellan --profile=tier_1_browsers,tier_2_browsersIf you're using profiles to reflect browser tiers in a large organization, you may wish to centralize your profiles on a web server somewhere and have magellan load them remotely. To do this, specify a profile URL to source profiles from, and the profile you want to use after a # (hash):
$ magellan --profile=http://example.com/testing/browser_profiles.json#tier2Multiple profiles can be specified the same way, comma delimited:
$ magellan --profile=http://example.com/testing/browser_profiles.json#tier1,tier2Where browser_profiles.json should have a structure similar to placing profiles{} in magellan.json:
{
"profiles": {
"tier_2": [
{ "browser": "safari_7_OS_X_10_9_Desktop" },
{ "browser": "IE_8_Windows_2008_Desktop" },
{ "browser": "IE_9_Windows_2008_Desktop" },
{ "browser": "IE_10_Windows_2008_Desktop" }
]
}If you have setup and teardown tasks that need to run before and after magellan is called (for compilation, standing up a mocking server, cleanup, etc), then you can expose the following npm task flow using the scripts example below:
$ npm run magellan:setup
$ magellan
$ npm run magellan:teardownand:
$ npm run magellan:setup
$ magellan --sauce
$ npm run magellan:teardownHere's an example scripts block that implements these tasks in your package.json:
"scripts": {
"magellan:setup": "./path/to/my/setup.sh",
"magellan:teardown": "./path/to/my/teardown.sh"
},All code not otherwise specified is Copyright Wal-Mart Stores, Inc. Released under the MIT License.