Interop 2024 Accessibility Investigation
cookiecrook opened this issue · 20 comments
Separate from the Interop 2024 Accessibility Focus Area, we'll also have an investigation area that doesn't contribute to the overall Interop 2024 score.
Scoring Criteria for Interop 2024 Accessibility Testing Investigation
[Significant Edit May 7, 2024] The Accessibility Investigation Group, in a meeting today and based on the culmination of discussion below, determined to abandon the Serialized Accessibility Tree Dump path in favor of a tree walker investigation using the Accessible Node direction. The 25% scoring criteria previously dedicated to Tree Dump is now repurposed into the Accessible Node investigation, including Tree Walking features.
-
(25%) Accessible Node or similar API
- (10%) Initial spec language agreement of Accessible Node (DRI @cookiecrook)
- WICG/aom#197
- and/or
- WICG/aom#203
- (10%) Initial implementation of Accessible Node (behind runtime flag?) in WebKit, Chromium, Gecko
- (5%) First Implementation in Gecko #1929144
- (3%) Second Implementation
- (2%) Third Implementation
- (5%) Tree Walker investigation using Accessible Node
- (3%) Write basic tree walker tests (
tentative) that purposefully avoid all known tree implementation differences (scroll areas, pseudo elements and other generated content, etc.) - (2%) Author investigation tests for which parts of Node or Tree might be standardized in 2025 or later?
- (1%) Purposefully write failing tests (
tentative) that expose the areas where the internal accessibility tree differ between engines - (1%) File AOM or ARIA issues if it appears are some tree relationships that might be standardized as part of a 2025 Investigation
- (1%) Purposefully write failing tests (
- (3%) Write basic tree walker tests (
- (10%) Initial spec language agreement of Accessible Node (DRI @cookiecrook)
-
(25%) Acacia AAM test exploration (DRI @spectranaut)
- (5%) initial prototype
- (5%) functional in implementations/APIs
- not calling out all combos, but as examples:
- Mac AX API in WebKit, Chromium, Gecko
- Linux GTK in Chromium, Gecko
- Windows UIA in Chromium, Gecko
- Others?
- Window MSAA? in Chromium, Gecko
- Window IA2? in Chromium, Gecko
- not calling out all combos, but as examples:
- (5%) RFC consensus?
- (5%) functional in WPT CI?
- (5%) additional test writing?
-
(30%) Documentation
- (5%) Initial call-for-feedback in Google Docs draft
- (25%) #83 (DRI @rahimabdi)
- (10%) Incorporate feedback from reviewers
- (15%) Deliver/publish final v1 standalone documentation
-
(20%) Miscellaneous
Carryover 2025 Investigation
Note: I tweaked the numbers above a bit since the time of the call this morning to account for the two items now marked as 3% each. I also added all the "candidate" issues to the final 10% item.
I'd like to understand the reason we're looking at prototyping both tree dump and Accessible Node. It seems to me that the concept of Accessible Node is more flexible: you can implement a tree dump framework using Accessible Node (by walking all nodes from JS, likely packaged into a nice utility module), whereas you can't really implement Accessible Node using a tree dump. Tree dump also seems like it might be less flexible and more challenged by the tree structure differences between engines, which, while mostly inconsequential to users and AT, will be very troublesome in tests. Is there some context written down somewhere that could help me understand better? Thanks.
As I understand it, Google & Bocoup wanted to prototype the tree dump for the dual purpose of 1. syncing WebDriver and CDP accessors to that feature, and 2. That it will allow some limited amount of WPT testing potentially more quickly the other path. For example, though the output trees may be slightly different, substring searches on leaf nodes could allow us to test things like the hidden state of certain nodes faster than the other path.
Since it's an investigation (not a Focus Area), I don't think there's any harm in exploring that path. Currently it's blocked because the investigators (@mzgoddard, @boazsender, etc) have not yet addressed the RFC guidance in this comment. If that happens, we can continue the exploration. If not, that's okay too. We would hit 100% We would NOT hit 100% of the investigation, but that's why it's an investigation.
Minor note: we are at 1% or 2%, so we could get the investigation score off the ground, but I'm not really worried about updating it until we hit the 5% or 10% mark.
@cookiecrook you mean we would not hit 100%, right?
From Gecko's point of view, since we don't already have a tree dump API, we're reluctant to invest time into implementing two things instead of one.
Responding to a few different threads in this issue:
Next steps for tree dumping:
After our chromium tree-dumping research last year, we think the next step should also be spec writing in the form of two standards:
- a web driver bidi extension for tree dumping
- a spec for a small part of the accessibility tree that each implementation can agree on the shape of, location TBD
We no longer think that landing the chromium prototype is the right next step. We did start an RFC conversation, which ultimately led to this new direction.
I think we should update the roadmap above to reflect this. Perhaps our 2024 scope should be the specs, and not the implementation.
@zcorpan would Gecko consider implementing tree dumping in 2025 if there were specs?
Next steps for accessible nodes:
For accessible nodes, the only comment we have is that we recommend specifying it as a bidi extension, rather than in webdriver classic.
Pursuing two testing paths:
I think the question of whether to pursue both accessible node and tree dumping strategies in the long term comes down to two things
- whether or not we think that more testing surface would lead to more stable and interoperable ARIA rendering in the platform.
- whether or not we think pursuing two paths would ultimately lead to a better single strategy
@jcsteh et al, do you think specing and testing parts of trees and along with accessible nodes would improve ARIA rendering interop?
A few replies to @boazsender:
a web driver bidi extension for tree dumping
As Gecko doesn't support tree dump functionality and WebKit doesn't support webdriver bidi, this path may not enable near-term testing in anything besides Chromium. IMO, if this is to kept on the Investigation roadmap, at least one of those variables should change to support testing this year in at least two implementations.
a spec for a small part of the accessibility tree that each implementation can agree on the shape of
I think AOM #197 & AOM #203 may be a more likely path toward standardizing... I thought the goal of the tree dump was effectively a temporary stand-in API that might enable testing sooner than the longer-term interoperable web driver APIs.
- accessibility tree that each implementation can agree on the shape of, location TBD
- whether or not we think that more testing surface would lead to more stable and interoperable ARIA rendering in the platform.
AOM or ARIA I think... but isn't that what AOM #203 would standardize? We know there are tree differences, so aligning parent/child returns .tentative WPT files using webdriver ~accessiblenode could be the way to detect (and eventually standardize) those differences. Once implementations pass all the ~accessiblenode tree walker tests consistently, then we could work on standardizing the tree dump functionality, right? Until then, substring tree dump tests (even if not identical) may provide short term benefit.
For accessible nodes, the only comment we have is that we recommend specifying it as a bidi extension, rather than in webdriver classic.
Since all engines support webdriver classic, isn't that the better near term approach? Nothing in the AOM #203 proposal prevents longer-term adoption into webdriver bidi. It even mentions certain features (live region testing, for example) should be deferred until bidi support is universal.
That it will allow some limited amount of WPT testing potentially more quickly the other path. For example, though the output trees may be slightly different, substring searches on leaf nodes could allow us to test things like the hidden state of certain nodes faster than the other path.
Why would this be faster with tree dump vs ~AccessibleNode? Either way, we will (for now) need to potentially fuzzy match things to make it work across browsers. As I understand it, either approach exposes a bunch of properties for a given node. It's just that tree dump exposes them for all nodes, whereas ~AccessibleNode exposes them for a single node (and you then walk to other nodes to query those).
~AccessibleNode would be known list of standardized, normalized properties that all implementations approve, not the entirety of each implementation's internal representation.
In contrast, the tree dump path would just be a standardized accessor to a serialized, non-standard, engine internal representation... In Chrome and WebKit, those internal tree accessors already exist as Accessibility.getFullAXTree and dumpAccessibilityTree. I assume Gecko has something similar, or something that could be converted without too much effort.
It'd be more work to normalized the tests to accept substring variants of the same info from different engines (hidden: true vs isHidden: YES perhaps), but lower implementation complexity, so hence it being referred to above as a "faster" path to making things testable.
I wrote this:
I assume Gecko has something similar, or something that could be converted without too much effort [into a serialized tree].
backtracking a bit since @zcorpan said above:
From Gecko's point of view, since we don't already have a tree dump API, we're reluctant to invest time into implementing two things instead of one.
Completely reasonable. In any case, the point is moot until someone makes progress on the Chrome prototype. If that ends up being abandoned (or if Gecko stakeholders feel strongly enough to object), we can pull it from the scoring criteria, too.
For what it's worth, my priority and focus is also on the ~AccessibleNode path.
Based on the group consensus in today's meeting, I've adjusted the scoring criteria significantly above... The most significant change (abandonment of the serialized tree dump) is called out specifically. Please comment, change, or otherwise let me know if you have any questions or concerns.
score updated: 20*(6/17) + 5 = 12%
Current score: 20*(6/13) + 15 = 24.231% 🎉
Mostly due to @rahimabdi's documentation progress.
Should we bring in @spectranaut and @alice's work on the Acacia AAM testing in the scoring criteria for the 2024 Accessibility Investigation? They made significant progress recently and including it would bring the RFC more visibility.
[Update: Aug 13, 2024] Recommend dropping the following from the Miscellaneous section unless a DRI steps up
We resolved to drop these in #140 (comment)
Should we bring in @spectranaut and @alice's work on the Acacia AAM testing in the scoring criteria for the 2024 Accessibility Investigation? They made significant progress recently and including it would bring the RFC more visibility.
Yes per #140 (comment)
Scoring Criteria updated per meeting resolution: Add Acacia, and drop the proposed DRI-less miscellaneous issues.
5 + 30 + 20*(7/13) = 45.769%
5 + 10 + 30 + 20*(7/13) = 55.769%