We use Protractor and Jasmine for our GUI E2E and Integration tests. However in the near future we are having to basically change our existing projects, because of the WebDriverJS promise manager is being to deprecate and remove. So we saw the time to look around and give something new a try.
The Depreciation Plan says:
- Phase 1: allow users to opt-out of the promise manager (October 2016)
- Phase 2: opt-in to the promise manager (October 2017)
- Phase 3: removing the promise manager (October 2018)
check the SeleniumHQ's github issue for details.
"Instead of the control flow, you can synchronize your commands with promise chaining or the upcoming ES7 feature async/await...The latest Node.js provides native async/await, which means we can get stable e2e test without using control flow in javascript test.
Note: To write and run native async/await test, the node.js version should be greater than or equal to 8.0, and Jasmine version should be greater than or equal to 2.7" by Protractor
- An Overview of JavaScript Testing in 2018
- Use React tools for better Angular apps
- UI Test Automation Frameworks Showdown: TestCafé versus Nightwatch.js
- End-to-End Testing with TestCafe
- TestCafe: Easier End-to-end Web App Testing with Node.js
| Categories | Protractor | WebDriverIO | Night Watch | TestCafe |
|---|---|---|---|---|
| Architecture | Protractor is a wrapper around WebdriverJS (selenium-webdriver node module). | WebdriverIO is custom implementation of W3C webdriver JSON wire protocol. | Night Watch is custom implementation of W3C webdriver JSON wire protocol. | TestCafe uses a URL-rewriting proxy instead of WebDriver. This proxy injects the driver script that emulates user actions into the tested page. This way TestCafe can do everything required for the testing: it can emulate user actions, authentication, run its own scripts & etc. |
| Supported browsers | Edge, Chrome, Firefox, Safari, Opera, HtmlUnit | Edge, Chrome, Firefox, Safari, Opera, HtmlUnit, PhantomJs | Edge, Chrome, Firefox, Safari, Opera, HtmlUnit, PhantomJs | Edge, Chrome, Firefox, Safari, Opera (works with any browser that supports HTML5 without any plugins) |
| Angular specific locator strategies | Supported | Partially supported by adding ‘webdriverjs-angular’ node module. Only ‘WaitForAngular’ has been implemented. | Not supported. But still we should be able to automate angular js application as selenium does. | Supported |
| Inbuilt test runner | Available | Available | Available | Available |
| Supported testing frameworks | Jasmine, Mocha, Cucumber | Jasmine, Mocha, Cucumber | Mocha, inbuilt framework | inbuilt framework |
| Reporting | Junit Xml Reporter, Protractor Html screenshot reporter, Jasmine spec reporter | Allure reporting, Jasmine Spec reporter, Junit xml reporter, Cucumber reporting | Junit xml reporter,Mocha reporting | spec,list,minimal,xUnit,JSON |
| Parallel Execution | Supported | Supported | Supported | Only Concurrency (is an optional mode that allows you to invoke multiple instances of the same browser. These instances constitute the pool of browsers against which tests run concurrently, i.e. each test runs in the first free instance.) |
| Cloud Execution (Sauce Labs, BrowserStack, Testing bot) | Supported | Supported | Supported | Supported |
| Mobile Support (Appium) | Partial support (Supports mobile browsers but does not support native apps) | Android, IOS | Android, IOS | Android, IOS (can run on mobile devices by default, without requiring custom configuration) |
| Synchronous execution | Supported | Supported | Supported | Supported |
| Page Object Pattern | Supported | Supported | Supported | Supported |
| Continuous Integration | Supported | Supported | Supported | Supported |
| Retrying flaky tests | Possible | Possible | Possible | Possible (With enabling the quarantine mode for tests that fail. In this mode, a failed test is executed several more times.) |
| TypeScript support | Inbuilt support | Possible when using Codecept.js | Not yet supported | Inbuilt support |
| Additional Features | Angular application support | CSS regression testing using ‘webdrivercss’ npm package. | – | Plug-in Free Testing (after install, immediately functional), nice async/await syntax |
| Licenses | MIT | MIT | MIT | MIT |
| Dependencies | 15 | 21 | 10 | 62 |
I created a new Protractor project as well to comparison. I applied the "new" Async/Await syntax (learn more at JavaScript Info). So you can easily compare the TestCafe and the Protractor way for the same application and goal.
Create a demo project with TestCafe to cover a cluster creation workflow with Cloudbreak GUI.
Test command in package.json:
"test": "node_modules/.bin/testcafe chrome tests/*.ts -r spec,xunit:result.xml -S -s results/screenshots --debug-on-fail",
- Set the needed environment variables for environment file
yarn test
Aside for Chrome, TestCafe supports a number of browsers out-of-the-box. Further, if you don’t need to test browser-dependent functionality, then you can use a headless browser.
Testing in Headless Mode I've introduced a new test command in package.json:
"test-headless": "node_modules/.bin/testcafe chrome:headless tests/loginTest.ts",
- Set the needed environment variables for environment file
yarn test-headless
"TestCafe provides a preconfigured TestCafe Docker image with Chromium and Firefox installed. Therefore, you can avoid manual installation of browsers and the testing framework on the server." by TestCafe
docker pull testcafe/testcafe
So with the help of this image you can run your locally developed and built tests in the related Docker container with headless Google Chrome and/or Firefox.
Unfortunately the official TestCafe Docker image is not sufficient for CI test execution, because of we need to install NPM components then build the project after a GitHub clone. So I created a new Docker image for this purpose, based on the TestCafe one.
Till this Docker image becomes an official Hortonworks image at DockerHub, you should build this:
make build
This command creates the aszegedi/testcafe (~730 MB) Docker image in your local Docker environment based on the project's Docker file.
- Set the needed environment variables for environment file
make run
This script is optimized for CI execution. So the Docker container starts only with -i operator.
- -t : Allocate a pseudo-tty
- -i : Keep STDIN open even if not attached
Page object is a design pattern that can be implemented as a Selenium best practices. The functionality classes (PageObjects) in this design represent a logical relationship between the pages of the application. There are other design patterns that also may be used in testing. Some use a Page Factory for instantiating their page objects. Discussing all of these is beyond the scope of this introduction. So I've just collected some basic thoughts and principles here for summary:
"A page object wraps an HTML page, or fragment, with an application-specific API, allowing you to manipulate page elements without digging around in the HTML." by Martin Fowler
"By introducing the “elements as first class citizens” principle it is now possible to build up large test suites using this pattern. There are no additional packages required to create page objects. It turns out that
Object.createprovides all necessary features we need:
- inheritance between page objects
- lazy loading of elements and
- encapsulation of methods and actions
The goal behind page objects is to abstract any page information away from the actual tests. Ideally you should store all selectors or specific instructions that are unique for a certain page in a page object, so that you still can run your test after you’ve completely redesigned your page." by WebDriverIO
"The basic rule of thumb:
- It should allow to do anything and see anything that a human can
- It should also provide an interface that's easy to access and modify
- It should hide the underlying widgetry
- It should have accessor methods that take and return your values and commands
- check boxes should use booleans
- buttons should be represented by action oriented method names
The page object should encapsulate the mechanics required to find and manipulate the data in the gui control itself." by Martin Fowler
"From the structural side it makes sense to separate spec files and page objects and put them into different directories."... ..."Note that you can build up way more complex page object structures than this. For example have specific page objects for modals or split up a huge page object into different objects that inherit from the main page object. The pattern gives you really a lot of opportunities to encapsulate page information from your actual tests, which is important to keep your test suites structured and clear in times where the project and number of tests grows." by WebDriverIO