andrewhick.com test suite
Cypress and Axe proof of concept with andrewhick.com
This test suite uses Cypress and Axe to test andrewhick.com for accessibility. This README goes through the steps to run these tests, and to set up a brand new test suite from scratch.
It assumes that you have a basic knowledge of:
- Cypress
- Terminal / command prompt
- GitHub
- Node Package Manager (npm)
Install and run this test suite
- Clone this repository to your local machine and navigate to the folder in terminal:
git clone https://github.com/andrewhick/andrewhick-test.git && cd andrewhick-test
npm install
npx cypress open
to open the Cypress test windownpx cypress run
to run from command line instead (best for Axe)
How to set up a brand new Cypress project with Axe
This guide covers how to set up a new project from scratch.
Example 'Hello world' test
This is the simplest possible test once your test suite is set up. It flags a full set of accessibility rules and does not provide detailed output to the terminal, just the number of violations on failure.
describe('Hello world', () => {
it('should be accessible', () => {
cy.visit('/')
cy.injectAxe()
cy.checkA11y()
})
})
Set up Git
Skip this section if you are not interested in publishing your repository to GitHub.
- Set up a new folder on your machine and navigate to it. The folder name will become your GitHub repository's name, so choose wisely.
- Add a README.md file
git init
git checkout -b main
git add .
git commit am "First commit"
git remote add origin https://github.com/andrewhick/andrewhick-test.git
(replace this with your GitHub username and test repository name)git push -u origin main
Install Cypress
Cypress installation instructions
npm install
npm init
npm install cypress --save-dev
npx cypress open
- this creates the folder structure the first time it is run, containing example tests.- Run test from command line using
npx cypress run
- Update cypress.json to include your base URL, set a default timeout, and prevent tests running every time tests are updated, to prevent unnecessary noise:
"baseUrl" : "https://www.andrewhick.com",
"defaultCommandTimeout" : 5000,
"watchForFileChanges": false
Install and set up Axe
Based on Tim Deschryver's tutorial plus extra information as needed.
npm i --save-development axe-core cypress-axe
installs the two main components- Add
import 'cypress-axe'
tosupport/index.js
- Add
cy.injectAxe()
to the test after page load - Add
cy.checkA11y()
when you want to test the page - Add a violation callback to give meaningful information in terminal
- Ask Axe to only flag the rules you want to test against, such as WCAG v2.1 levels A and AA
// Checks for accessibility against WCAG 2.1 A and AA, and outputs a meaningful response back to the terminal
cy.checkA11y(
null,
{
runOnly: {
type: 'tag',
values: ['wcag21a', 'wcag21aa']
}
},
terminalLog
)
By this stage you should hopefully have a basic, working test suite. Run it with npx cypress run
and check the outputs.
Convert shared functions to custom commands
If you plan to reuse functions such as the violation callback and your customised cy.checkA11y()
function, then you will save time and make your tests more readable by converting them to custom commands.
See /cypress/support/commands.js to see how they have been implemented here.
Pitfalls
1 accessibility violation was detected: expected 1 to equal 0
This just means the number of accessibility issues was 1 but should have been 0.
To make Axe produce a more meaningful report in the terminal you can use the violation callback.
I changed this by adding it as a custom command into support/commands.js
, and calling it back from cy.checkA11y
as cy.terminalLog
.
This is what a report looks like:
┌─────────┬─────────────────────┬────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────┐
│ (index) │ id │ impact │ description │ nodes │
├─────────┼─────────────────────┼────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────┤
│ 0 │ 'landmark-one-main' │ 'moderate' │ 'Ensures the document has only one main landmark and each iframe in the page has at most one main landmark' │ 1 │
│ 1 │ 'region' │ 'moderate' │ 'Ensures all page content is contained by landmarks' │ 1 │
└─────────┴─────────────────────┴────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────┘
Axe bug logged and closed for '1=0' error (someone else had the same issue as me and didn't understand the output)
Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "script-src 'self' 'unsafe-inline'".
This occurred because Cypress did not recognise the Content Security Policy (CSP) I'd set in the meta tags on my website. Instead they need to be set at server level.
Based on advice when I asked about it on StackOverflow, I decided to comment out the CSP meta tag on my website and the tests ran again.
Non-WCAG rules were still included in scope
I tried setting rules to only test for WCAG 2.1 A and AA as follows:
// Tags were not successfully applied:
cy.configureAxe({
rules: [
{
id: 'WCAG 2.1',
tags: ['wcag21a', 'wcag21aa'],
disableOtherRules: true
}
]
})
But this still seemed to flag up landmark violations, which aren't part of WCAG 2.1 A/AA. I don't know why yet, but I found another solution which worked:
Instead, skipping the configureAxe step and applying the config directly to cy.checkA11y()
worked for me:
// This worked and only flagged WCAG rules:
cy.checkA11y(
null,
{
runOnly: {
type: 'tag',
values: ['wcag21a', 'wcag21aa']
}
},
terminalLog
)
Useful links
- Documentation on cy.checkA11y()
- NPM documentation on cypress-axe
- Axe rules configuration
- List of available rules tags
- Example config (not yet used successfully!)
- Documentation on axe.configure (not yet used successfully!)