/browserstack-examples-nightwatchjs

This repository showcases the BrowserStack integration features with the NightWatchJS using the BrowserStack sample demo canonical app.

Primary LanguageJavaScript

Logo

BrowserStack Examples Nightwatchjs Node.js NightWatch

Introduction

Nightwatch.js is an integrated, easy to use End-to-End testing solution for web applications and websites, written in Node.js. It uses the W3C WebDriver API to drive browsers in order to perform commands and assertions on DOM elements.

This BrowserStack Example repository demonstrates a Selenium test framework written in Node.js and Nightwatchjs 1.5 with parallel testing capabilities. The Selenium test test scripts are written for the open source BrowserStack Demo web application (Github). This BrowserStack Demo App is an e-commerce web application which showcases multiple real-world user scenarios. The app is bundled with offers data, orders data and products data that contains everything you need to start using the app and run tests out-of-the-box.

The Selenium test tests are run on different platforms like on-prem, docker and BrowserStack using various run configurations and test capabilities.

Repository setup

  • Clone the repository

  • Ensure you have the following dependencies installed on the machine

    • Node.js >= 10.17.0
    • yarn >= 1.22

  • To install repository dependencies run:

    yarn install

About the tests in this repository

This repository contains the following Selenium tests:

Module Test name Description Tag
E2E End to End Scenario This test scenario verifies successful product purchase lifecycle end-to-end. It demonstrates the Page Object Model design pattern and is also the default test executed in all the single test run profiles. e2e
Login Login with given username This test verifies the login workflow with different types of valid login users. login
Login Login as Locked User This test verifies the login workflow error for a locked user. login
Offers Offers for Mumbai location This test mocks the GPS location for Mumbai and verifies that the product offers applicable for the Mumbai location are shown. offers
Product Apply Apple & Samsung Vendor Filter This test verifies that only Apple and Samsung products shown if the Apple and Samsung vendor filter option is applied. product
Product Apply Lowest to Highest Order By This test verifies that the product prices are in ascending order when the product sort "Lowest to Highest" is applied. product
User Login as User with no image loaded This test verifies that the product images load for user: "image_not_loading_user" on the e-commerce application. Since the images do not load, the test case assertion fails. user
User Login as User with existing Orders This test verifies that existing orders are shown for user: "existing_orders_user" user

Test infrastructure environments

Configuring the maximum parallel test threads for this repository

For all the parallel run configuration profiles, you can configure the maximum parallel test threads by changing the settings below.

  • Docker

    resources/conf/runners/docker.js

    module.exports = {
 ...

 test_workers: {
   ...
  workers: 5, // change this attribute
   ...
 },
...

  • BrowserStack

    resources/conf/runners/browserstack.js

    module.exports = {
 ...

 test_workers: {
   ...
  workers: 5, // change this attribute
   ...
 },
...

Test Reporting

On Premise

This infrastructure points to running the tests on your own machine using a browser (e.g. Chrome) using the browser's driver executables (e.g. ChromeDriver for Chrome). Selenium enables this functionality using WebDriver for many popular browsers.

Om-Prem Prerequisites

  • For this infrastructure configuration (i.e on-premise), ensure that the ChromeDriver is downloaded successfully via node_modules

Note: The ChromeDriver version must match the Chrome browser version on your machine.

Running Your Tests

Run a specific test on your own machine

  • How to run the test?

    To run the default test scenario (e.g. End to End Scenario) on your own machine, use the following command:

    yarn on-prem-single

    To run a specific test file, use the following command with the additional 'tag-name' argument:

    yarn on-prem --tag <tag-name>
# for eg: yarn on-prem --tag user

    where, the argument 'tag-name' can be any tag configured in this repository.

    E.g. "user", "login" or any other tag as outlined in About the tests in this repository section.

  • Output

    This run profile executes a specific test scenario on a single browser instance on your own machine.

Run the entire test suite on your own machine

  • How to run the test?

    To run the entire test suite on your own machine, use the following command:

    yarn on-prem-suite

  • Output

    This run profile executes the entire test suite sequentially on a single browser, on your own machine.

Docker

Docker is an open source platform that provides the ability to package and test applications in an isolated environment called containers.

Docker Prerequisites

  • Install and start Docker.
  • Note: Docker should be running on the test machine. Ensure Docker Compose is installed as well.
  • Run docker-compose pull from the current directory of the repository.

Running Your Tests

Run a specific test on the docker infrastructure

  • How to run the test?

    • Start the Docker by running the following command:
    docker compose --project-directory docker up
    • To run the default test scenario (e.g. End to End Scenario) on your own machine, use the following command:
    yarn docker-single

    To run a specific test file, use the following command with the additional 'tag-name' argument:

    yarn docker --tag <tag-name>
# yarn docker --tag user

    where, the argument 'tag-name' can be any tag configured in this repository.

    E.g. "user", "login" or any other tag as outlined in About the tests in this repository section.

    • After tests are complete, you can stop the Docker by running the following command:
    docker compose --project-directory docker down

  • Output

    This run profile executes a specific test scenario on a single browser deployed on a docker image.

Run the entire test suite in parallel using Docker

  • How to run the test?

    • Start the docker image first by running the following command:
    docker compose --project-directory docker up -d
    • To run the entire test suite in parallel on the docker image, use the following command:
    yarn docker-parallel
    • After the tests are complete stop the Selenium grid by running the following command:
    docker compose --project-directory docker down

  • Output

    This run profile executes the entire test suite in parallel on a single browser, deployed on a docker image.

  • Note: By default, this execution would run maximum 2 test threads in parallel on Docker. Refer to the section "Configuring the maximum parallel test threads for this repository" for updating the parallel thread count based on your requirements.

BrowserStack

BrowserStack provides instant access to 2,000+ real mobile devices and browsers on a highly reliable cloud infrastructure that effortlessly scales as testing needs grow.

BrowserStack Prerequisites

  • Create a new BrowserStack account or use an existing one.

  • Identify your BrowserStack username and access key from the BrowserStack Automate Dashboard and export them as environment variables using the below commands.

    • For *nix based and Mac machines:
    export BROWSERSTACK_USERNAME=<browserstack-username> &&
export BROWSERSTACK_ACCESS_KEY=<browserstack-access-key>
    • For Windows:
    set BROWSERSTACK_USERNAME=<browserstack-username>
set BROWSERSTACK_ACCESS_KEY=<browserstack-access-key>

    Alternatively, you can also hardcode username and access_key objects in the browserstack.json file.

Note:

  • We have configured a list of test capabilities in the browserstack.json file. You can certainly update them based on your device / browser test requirements.
  • The exact test capability values can be easily identified using the Browserstack Capability Generator

Running Your Tests

Run a specific test on BrowserStack

In this section, we will run a single test on Chrome browser on Browserstack. To change test capabilities for this configuration, please refer to the single object in caps.json file.

  • How to run the test?

    • To run the default test scenario (e.g. End to End Scenario) on your own machine, use the following command:
    yarn bstack-single

    To run a specific test file, use the following command with the additional 'tag-name' argument:

    yarn bstack --tag <tag-name>
# for eg yarn bstack --tag user

    where, the argument 'tag-name' can be any tag configured in this repository.

    E.g. "user", "login" or any other tag as outlined in About the tests in this repository section.

  • Output

    This run profile executes a single test on a single browser on BrowserStack. Please refer to your BrowserStack dashboard for test results.

Run the entire test suite in parallel on a single BrowserStack browser

In this section, we will run the tests in parallel on a single browser on Browserstack. Refer to single object in caps.json file to change test capabilities for this configuration.

  • How to run the test?

    To run the entire test suite in parallel on a single BrowserStack browser, use the following command:

    yarn bstack-parallel

  • Output

    This run profile executes the entire test suite in parallel on a single BrowserStack browser. Please refer to your BrowserStack dashboard for test results.

Run the entire test suite in parallel on multiple BrowserStack browsers

In this section, we will run the tests in parallel on multiple browsers on Browserstack. Refer to the parallel object in caps.json file to change test capabilities for this configuration.

  • How to run the test?

    To run the entire test suite in parallel on multiple BrowserStack browsers, use the following command:

    yarn bstack-parallel-multiple

[Web application hosted on internal environment] Running your tests on BrowserStack using BrowserStackLocal

Prerequisites

  • Clone the BrowserStack demo application repository.

    git clone https://github.com/browserstack/browserstack-demo-app

  • Please follow the README.md on the BrowserStack demo application repository to install and start the dev server on localhost.

  • In this section, we will run a single test case to test the BrowserStack Demo app hosted on your local machine i.e. localhost. Refer to the <browser/device>_local object in caps.json file to change test capabilities for this configuration.

  • Note: You may need to provide additional BrowserStackLocal arguments to successfully connect your localhost environment with BrowserStack infrastructure. (e.g if you are behind firewalls, proxy or VPN).

  • Further details for successfully creating a BrowserStackLocal connection can be found here:

[Web application hosted on internal environment] Run a specific test on BrowserStack using BrowserStackLocal

  • How to run the test?

    • To run the default test scenario (e.g. End to End Scenario) on a single BrowserStack browser using BrowserStackLocal, use the following command:
    yarn bstack-local

    To run a specific test file, use the following command with the additional 'tag-name' argument:

    node ./resources/conf/runners/browserstack_local.js -c resources/conf/runners/browserstack.js --tag <tag-name>

    where, the argument 'tag-name' can be any tag configured in this repository.

    E.g. "user", "login" or any other tag as outlined in About the tests in this repository section.

  • Output

    This run profile executes a single test on an internally hosted web application on a single browser on BrowserStack. Please refer to your BrowserStack dashboard(https://automate.browserstack.com/) for test results.

[Web application hosted on internal environment] Run the entire test suite in parallel on a single BrowserStack browser using BrowserStackLocal

In this section, we will run the test cases to test the internally hosted website in parallel on a single browser on Browserstack. Refer to the chrome object in caps/browserstack.json file to change test capabilities for this configuration.

  • How to run the test?

    To run the entire test suite in parallel on a single BrowserStack browser using BrowserStackLocal, use the following command:

    yarn bstack-local-parallel

  • Output

    This run profile executes the entire test suite on an internally hosted web application on a single browser on BrowserStack. Please refer to your BrowserStack dashboard for test results.

  • Note: By default, this execution would run maximum 10 test threads in parallel on BrowserStack. Refer to the section "Configuring the maximum parallel test threads for this repository" for updating the parallel thread count based on your requirements.

[Web application hosted on internal environment] Run the entire test suite in parallel on multiple BrowserStack browser using BrowserStackLocal

In this section, we will run the test cases to test the internally hosted website in parallel on multiple browsers on Browserstack. Refer to the <browser/device>_local object in caps/browserstack.json file to change test capabilities for this configuration.

  • How to run the test?

    To run the entire test suite in parallel on a single BrowserStack browser using BrowserStackLocal, use the following command:

    yarn bstack-local-parallel-multiple

  • Output

    This run profile executes the entire test suite on an internally hosted web application on multiple browsers on BrowserStack. Please refer to your BrowserStack dashboard for test results.

  • Note: By default, this execution would run maximum 10 test threads in parallel on BrowserStack. Refer to the section "Configuring the maximum parallel test threads for this repository" for updating the parallel thread count based on your requirements.

Generating Allure Reports

  • Generate Report using the following command: yarn allure:generate
  • Serve the Allure report on a server: yarn allure:open

Additional Resources

Observations

  • Do not use "." in the naming convention for any browsers or devices as it might break nightwatchjs' engine to start sessions or run them correctly.
  • There are few occurrences of pause method being used in some of the scripts as a temporary measure to avoid flaky test execution. We plan to remove all occurrences of these hardcoded pause methods in the upcoming patch releases.

Open Issues

  • When running all the tests in parallel, there is some flakiness observed in mobile devices where in some commands are skipped or not executed at all.
  • Currently Nightwatch does not allow to control the maximum threads that should be spawned for tests running across different browsers in parallel. So when you run yarn bstack-local-parallel-multiple and yarn bstack-parallel-multiple commands, it will spawn 25 test sessions (5 spec files across 5 browsers).