Discuss - expose results/errors to callers?

[gabalafou]

I would like to start a discussion about the best way to address a use case that I think could be fairly common.

I am currently working on a project to run automated accessibility check against JupyterLab. For each app view (page) for which the checker detects violations, we want to upload not only the

1. nice human-friendly HTML version of the report, but also the
2. raw JSON used to generate the HTML report. We also want to upload a
3. snapshot of the accessibility tree.

With the current expect-axe-playwright code, I did not see any way for the caller (me) to get the results of the axe-core scan, so I forked the code and modified it to upload all three files whether or not the test fails.

One possible way forward would be to merge my code changes and allow the caller to toggle each upload on or off in the options passed to toBeAccessible. We would also need an option to toggle whether the uploads are done on success too.

expect(page).toBeAccessible({
   uploadOnSuccess: true,
   uploadHtmlReport: true,
   uploadJsonResults: true,
   uploadAccessibilityTree: true,
});

But I also wonder if there is a more general-purpose solution that is also idiomatic with respect to Expect.js.

For example, perhaps this library could support an alternative usage that breaks apart the Axe report generation from the expect call:

import { getAxeResults } from 'expect-axe-playwright';
const axeResults = getAxeResults(page, /* axeOptions */);
expect(axeResults).toHaveNoViolations();

Perhaps this reveals that main reason I wanted to use this library was for the way it integrates axe-core results with the Playwright test runner. It's nice to be able to scan several different pages for accessibility violations in a Playwright test suite and get readable, intelligible output for each tested page where there are violations.

What are your thoughts? I would be happy to do the work of writing code and tests if we come to a solution for my use case.

[mskelton]

Good discussion, while not something we need right now, I definitely can understand the reasons you present for why this would be helpful.

Some thoughts on your recommendations:

1. Add a reports option to toBeAccessible which would accept an array of report names (e.g. reports: ['html', 'json', raw']). This option could also be configured globally via axeOptions in playwright.config.ts
2. Totally good with exporting a function, perhaps runAxe. I would expect it would inherit axeOptions from playwright.config.ts just like toBeAccessible currently.
3. Not sure about a new toHaveNoViolations matcher, what if instead we add support to the toBeAccessible to also accept an axe results object? Then you could do this:

   import { runAxe } from 'expect-axe-playwright'
   const axeResults = runAxe(page)
   expect(axeResults).toBeAccessible()

[gabalafou]

Thanks so much for getting back to me and being open to making changes to this nice library!

I have opened three separate pull requests to ground this discussion in actual code.

• #22
• #23
• #24

[mskelton]

@gabalafou Version 3.0 released. Really appreciate your help!

https://github.com/Widen/expect-axe-playwright/releases/tag/v3.0.0