/andrewhick-test

Tests for andrewhick.com

Primary LanguageJavaScript

andrewhick.com test suite

Cypress and Axe proof of concept with andrewhick.com

This test suite uses Cypress and Axe to test andrewhick.com for accessibility. This README goes through the steps to run these tests, and to set up a brand new test suite from scratch.

It assumes that you have a basic knowledge of:

  • Cypress
  • Terminal / command prompt
  • GitHub
  • Node Package Manager (npm)

Install and run this test suite

  1. Clone this repository to your local machine and navigate to the folder in terminal:
git clone https://github.com/andrewhick/andrewhick-test.git && cd andrewhick-test
  1. npm install
  2. npx cypress open to open the Cypress test window
  3. npx cypress run to run from command line instead (best for Axe)

How to set up a brand new Cypress project with Axe

This guide covers how to set up a new project from scratch.

Example 'Hello world' test

This is the simplest possible test once your test suite is set up. It flags a full set of accessibility rules and does not provide detailed output to the terminal, just the number of violations on failure.

describe('Hello world', () => {
  it('should be accessible', () => {
    cy.visit('/')
    cy.injectAxe()
    cy.checkA11y()
  })
})

Set up Git

Skip this section if you are not interested in publishing your repository to GitHub.

Karl Broman's tutorial

  1. Set up a new folder on your machine and navigate to it. The folder name will become your GitHub repository's name, so choose wisely.
  2. Add a README.md file
  3. git init
  4. git checkout -b main
  5. git add .
  6. git commit am "First commit"
  7. git remote add origin https://github.com/andrewhick/andrewhick-test.git (replace this with your GitHub username and test repository name)
  8. git push -u origin main

Install Cypress

Cypress installation instructions

  1. npm install
  2. npm init
  3. npm install cypress --save-dev
  4. npx cypress open - this creates the folder structure the first time it is run, containing example tests.
  5. Run test from command line using npx cypress run
  6. Update cypress.json to include your base URL, set a default timeout, and prevent tests running every time tests are updated, to prevent unnecessary noise:
    "baseUrl" : "https://www.andrewhick.com",
    "defaultCommandTimeout" : 5000,
    "watchForFileChanges": false

Install and set up Axe

Based on Tim Deschryver's tutorial plus extra information as needed.

  1. npm i --save-development axe-core cypress-axe installs the two main components
  2. Add import 'cypress-axe' to support/index.js
  3. Add cy.injectAxe() to the test after page load
  4. Add cy.checkA11y() when you want to test the page
  5. Add a violation callback to give meaningful information in terminal
  6. Ask Axe to only flag the rules you want to test against, such as WCAG v2.1 levels A and AA
    // Checks for accessibility against WCAG 2.1 A and AA, and outputs a meaningful response back to the terminal
    cy.checkA11y(
      null,
      {
        runOnly: {
          type: 'tag',
          values: ['wcag21a', 'wcag21aa']
        }
      },
      terminalLog
    )

By this stage you should hopefully have a basic, working test suite. Run it with npx cypress run and check the outputs.

Convert shared functions to custom commands

If you plan to reuse functions such as the violation callback and your customised cy.checkA11y() function, then you will save time and make your tests more readable by converting them to custom commands.

See /cypress/support/commands.js to see how they have been implemented here.

Pitfalls

1 accessibility violation was detected: expected 1 to equal 0

This just means the number of accessibility issues was 1 but should have been 0.

To make Axe produce a more meaningful report in the terminal you can use the violation callback.

I changed this by adding it as a custom command into support/commands.js, and calling it back from cy.checkA11y as cy.terminalLog.

This is what a report looks like:

┌─────────┬─────────────────────┬────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────┐
│ (index) │         id          │   impact   │                                                 description                                                 │ nodes │
├─────────┼─────────────────────┼────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────┤
│    0    │ 'landmark-one-main' │ 'moderate' │ 'Ensures the document has only one main landmark and each iframe in the page has at most one main landmark' │   1   │
│    1    │      'region'       │ 'moderate' │                            'Ensures all page content is contained by landmarks'                             │   1   │
└─────────┴─────────────────────┴────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────┘

Axe bug logged and closed for '1=0' error (someone else had the same issue as me and didn't understand the output)

Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self' 'unsafe-inline'".

This occurred because Cypress did not recognise the Content Security Policy (CSP) I'd set in the meta tags on my website. Instead they need to be set at server level.

Based on advice when I asked about it on StackOverflow, I decided to comment out the CSP meta tag on my website and the tests ran again.

Non-WCAG rules were still included in scope

I tried setting rules to only test for WCAG 2.1 A and AA as follows:

    // Tags were not successfully applied:
    cy.configureAxe({
      rules: [
        {
          id: 'WCAG 2.1',
          tags: ['wcag21a', 'wcag21aa'],
          disableOtherRules: true
        }
      ]
    })

But this still seemed to flag up landmark violations, which aren't part of WCAG 2.1 A/AA. I don't know why yet, but I found another solution which worked:

Instead, skipping the configureAxe step and applying the config directly to cy.checkA11y() worked for me:

    // This worked and only flagged WCAG rules:
    cy.checkA11y(
      null,
      {
        runOnly: {
          type: 'tag',
          values: ['wcag21a', 'wcag21aa']
        }
      },
      terminalLog
    )

Useful links