pa11y/pa11y-ci

"level" configuration appears to be ignored

cadecairos opened this issue · 3 comments

cadecairos commented

According to the pa11y documentation you should be able to pass level in your configuration file to support a few different use-cases:

  • error: exit with a code of 2 on errors only, exit with a code of 0 on warnings and notices
  • warning: exit with a code of 2 on errors and warnings, exit with a code of 0 on notices
  • notice: exit with a code of 2 on errors, warnings, and notices
  • none: always exit with a code of 0

This does not appear to be respected by pa11y-ci: https://github.com/pa11y/pa11y-ci/blob/master/bin/pa11y-ci.js#L94-L98

In our use-case, we'd like to configure pa11y-ci to with level set to "none" so that we can view errors in accessibility, fix them, and once we hit 0, then change level to "error"

braican commented

I'm having the same issue. I've added the "level" configuration to my .pa11yci file:

{
  "defaults": {
    "timeout": 5000,
    "level": "none"
  },
  "urls": [
    "http://localhost:5000",
    "http://localhost:5000/button.html"
  ]
}

Everything runs fine, but when when I run an npm script I've set up to run the pa11y tests, the npm script is still indicating that the pa11y-ci command exited with a status code of 2:

➜ npm run test:pa11y

> @xxxxxx/project@0.0.1 test:pa11y
> pa11y-ci

Running Pa11y on 2 URLs:
 > http://localhost:5000/button.html - 1 errors
 > http://localhost:5000 - 1 errors

Errors in http://localhost:5000/button.html:

 • This element has insufficient contrast at this conformance level. Expected a contrast ratio of at least 4.5:1, but text in this element has a contrast ratio of 1.61:1.
   Recommendation:  change text colour to #767676.

   (html > body > button)

   <button class="button">A coolish button</button>

Errors in http://localhost:5000:

 • The html element should have a lang or xml:lang attribute which describes the language of the document.

   (html)

   <html><head> <meta charset="utf-8...</html>

✘ 0/2 URLs passed
npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! @xxxxxx/project@0.0.1 test:pa11y: `pa11y-ci`
npm ERR! Exit status 2
npm ERR! 
npm ERR! Failed at the @xxxxxx/project@0.0.1 test:pa11y script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/braican/.npm/_logs/2020-01-14T02_12_43_394Z-debug.log

All things considered, this isn't a huge deal for us; we're using pa11y to run mostly informal informational tests on individual components and we can probably run pa11y as a one-off script. But it would be nice to wrap our tests into a larger suite of scripts, and the npm error being thrown (while it makes sense, given the pa11y error code) can cause some confusion since it seems like the script itself is failing, rather than the accessibility tests.

pcolmer commented

According to the documentation for pa11y:

Note this configuration is only available when using Pa11y on the command line, not via the JavaScript Interface.

mandrasch commented

Has anybody found a robust solution for this? I thought about setting the threshold to 999999 which will avoid exit code 2, but then not all the errors are listed in the JSON report file ... 🤷