microsoft/accessibility-insights-service

[Bug] Combined report has the same rules in Failed, N/A and Passed

Closed this issue · 2 comments

ferBonnin commented

Describe the bug

In the combined report, some rules appear twice in different sections which makes it confusing to users.

Area

  • Install script
  • Update script
  • Service
  • CLI

To Reproduce

Steps to reproduce the behavior:

  1. Run a combined report on a large number of pages with different attributes
  2. notice how some rules would appear in Failed and N/A or Failed and Passed

Expected behavior

Rules should only appear in one of the three sections.

  • first in failed
  • if not failed but applicable, then pass
  • if not failed and are N/A then in N/A

Additional context

found via user feedback

RobGallo commented

@ferBonnin , Isn't it possible for a rule to have some instances which pass and some instances which fail? And wouldn't it then make sense to have the rules listed under both sections?

ferBonnin commented

@ferBonnin , Isn't it possible for a rule to have some instances which pass and some instances which fail?

Yes

And wouldn't it then make sense to have the rules listed under both sections?

No, in those cases, in the individual page report we only list it in fail; the expectation is to do the same thing with the combined report. From a user, they told us they look at pass when they previously had a rule that failed and want to check it is now fixed. Having them in both pass and fail, or fail and N/A, is confusing for users per feedback from users' interviews.