QAccessibleTextInterface support for edbee
Closed this issue · 42 comments
Edbee is not marked as a rich text widget for QAccessible (or a text widget at all), so when blind users put their focus inside it, it captures their cursor and they're not aware why.
It would be nice if the editor was hooked up to the QAccessibleTextInterface so screenreaders on linux/macos/windows could use edbee. We've just went through the process ourselves for our custom text rendering widget, and I think Qt has an example of an implementation in their source code too.
Didn't know this interface. I will look into it!
Thanks for the suggestion and the link to the Mudlet commit.
To comment further on this, there is no feedback whatsoever when keyboard focus enters the edbee widget. This means that while typing is actually being registered ( as a blind screne reader user, I can verify this using OCR ) text cannot be reviewed in any way other than using OCR, which is error prone and not suitable for working with code.
As an example of what I mean, I can tab through Mudlet's editor controls just fine, however, when I tab into the Edbee widget, all keyboard functionality appears to be lost. Of course, it is not lost, it's simply that Edbee has gained keyboard focus and is performing as it should, but there is no screen reader feedback other than, "Grouping", when this occurs.
Ideally, we should be able to use the arrow keys to review the text, and use SHIFT+arrows to make selections as is standard practice in text/code editors and word processors.
How do other text editors handle tab key though? Because on the one hand, you may want to tab to the next object in the window, but on the other hand you want to write a tabulator whitespace. Maybe put the latter mode while writing, then press Escape to stop writing and come back to the first mode? 🤔
The fact that pressing tab inserts a tab isn't really an issue, because CTRL+Tab breaks out of the editor. The issue is that nothing inside the editor is being read out, with the exception of the autocomplete menu.
We're interested in accelerating this as well as giving something back to Edbee for the years of using it at the same time - and would like to place a $1000 USD bounty on this. Would you like to take it up @gamecreature?
Hi @vadi2 I didn't know it had high priority for you.
I've been quite busy with work related things last couple of weeks. I've looked at the issue, but didn't start with it yet:
The direct Qt docs weren't very clear how to implement this. Diving in deeper required more time and effort, which wasn't available at that time.
Currently I'm enjoying my vacation, which maybe the perfect time to look at it more closely.
Btw Is the Mac OS X text reader function, good enough to test this functionality?
Nice ok. Let me figure out the requirements and I'll come back to you 👍
@vadi2 The last couple of days, I've been experimenting with QAccessibility, it isn't as beautiful as the implementation in VSCode, bus basic support t seems to work on my Mac. It still is a very rough version, haven't tested all implementation details of the interface.
Multiple selection support (like adding caret etc) isn't picked up by the screenreaders. (Don't know if QT is capable of doing this). I don't know how to send an signal/event to notify a caret/selection has been added. (though a signal is send the selection has been changed)
VSCode takes a very different approach: it seems to break the document into multiple child-accessibility items (lines), and later when moving the lines are split into characters-childs.
Ah that is a difficult one. The one thing I'm struggling with is also a limitation of QAccessibility - that I think affects mostly NVDA on Windows - is that blank lines cause issues and misreporting: https://bugreports.qt.io/browse/QTBUG-105035.
We dealt with it by giving players options to either delete or replace blank lines with a space... but that's not really feasible in a code editor. This will be a problem that needs working around somehow. Maybe reporting a space instead of a blank line would be good enough?
Btw Is the Mac OS X text reader function, good enough to test this functionality?
It's an excellent development tool, gets you most of the way there, but in the end we need windows and linux support too. Those can be tested in a VM though.
I will test it on my Windows VM. I'm curious if it will have the same problems as mentioned in the QT issue you mentioned.
Try it with NVDA (open-source reader), first with a Qt demo app in Qt Creator - it'll show the problem.
Currenly I'm debugging on Windows. Some progress, Windows Speaker is reading the text, the NVDA reader unfortunately doesn't.
Tricky part is, the edbee-widget consists out of several widgets. The TextEditorComponent is getting the keyboard-focus, though the TextEditorWidget is handling the Acessibility.
Need to figure how to make NVDA handle the correct component. Perhaps I need to do something with Focus events etc.
The windows speaker, now renders the focused lines and letters correctly.
Btw. I also have the problem of the empty lines being interpreted incorrectly.
Yeah, setting the keyboard focus is important - we've had to do a few hacks to make it work right, try this code: https://github.com/Mudlet/Mudlet/blob/development/src/TConsole.cpp#L2045-L2053 it should work for Narrator and NVDA.
Btw. I also have the problem of the empty lines being interpreted incorrectly.
That's the main thing I'm trying to figure out first before posting the bounty, it might be a blocker so I didn't want to waste your time...
Just as an idea if you report blank lines as a single space instead, does it work?
@vadi2, I'm currently figuring out why NVDA doesn't work (on Windows). I've updated to the latest Qt and NVDA versions.
The application receives similar QAccessibleTextSelectionEvent events. But it doesn't read the text.
Microsoft's reader works very good (even highlights lines and letters like VSCode).
The OS X reader also works as it should.
When I'm running NVDA, event the QtCreator code-editor isn't read like it should.
I don't really know why and what to do about it. It's a Qt / NVDA thing?
For now I will go back on the empty line-reading issue, and see if I can build a workaround for this.
After I will try to build/run on the Linux VM
For NVDA, there's a mode that can show visually what window and element is selected. Try enabling that - maybe it doesn't think edbee is in focus?
@vadi2, NVDA is working now. 😃 I needed to switch the base-component to the TextEditorComponent instead of the TextEditorWidget. (Tried that before, but this time fixed the event so the correct widget is send with the events. 🤦)
It btw still has the empty line read bug. (I tried solving it in the text method, without success). But I guess I need to implement the textAfterOffset, textAtOffset and textBeforeOffset methods to build a workaround for it.
Did some extended debugging with AccessibleTextEditorWidget and trying to fix it with textAtOffset function.
But without success.
Somehow the Accessibility interface seems to call the text(..) method to get the full text, and does something with this.
It simply doesn't play nice with empty lines.
(When I perform a replace of "\n\n" => "\n \n in the text(..) method, the reader seems to read the lines correctly, but offsets, are going bogus further in document of course)
An alternative (but pretty hard) workaround solution, I could try to make is a virtual 'AccessibilityTextDocument'. (decorator kind of document-interface)
Which adds a space to empty lines and adjusts the offsets and modifies the TextDocument methods to adjust according to this 'virtual character'.
This probably works for screen readers, but I don't have a clue if the editableText Accessibility interface can cope with it.
Btw I tried to figure out what happens in the Qt code. Didn't find a direct cause. Though I have a suspicion, there are 2 kinds of boundary types sent with the Accessibility interface. Qt treats these types as one.
QAccessible::TextBoundaryType AtSpiAdaptor::qAccessibleBoundaryType(int atspiTextBoundaryType) const
..
case ATSPI_TEXT_BOUNDARY_SENTENCE_START:
case ATSPI_TEXT_BOUNDARY_SENTENCE_END:
return QAccessible::SentenceBoundary;
..
}
``The windows experience has improved by adding virtual spaces before the newlines.
This 'solves' the empty line reading bug.
There are still two know issues open: (
- selection recangles are calculated (and drawn) incorrectly when line endings are included.
(The last returned character is the first character of the next line because of the virtual space. The square drawing algorithm takes the end position of the last char.) - At the end of the document the line is read incorrectly, perhaps we need to add a virtual trailing space somehow.
Here's a screen capture that compares the reading output of QTextEdit vs Edbee:
Schermopname.2022-09-02.om.09.50.21.mp4
The windows experience has improved by adding virtual spaces before the newlines.
Brilliant! Great that you found a fix for this.
Thinking of the bounty requirements then - how about when remaining issues are ironed out and we get 2 visually impaired users sign off on usability, bounty will be complete and we'll pay you via Paypal (that's where the funds are). Does that sound fair?
By the way, NVDA has a Speech log feature - I found enabling that helpful for my sanity while developing this. It'll print the text it is saying in a window.
"By the way, NVDA has a Speech log feature..."
Thanks! Should have know that earlier, at the moment the computer's voice is continuing in my head. .. 😂
I think calling it a fix is not correct. It's a pretty dirty workaround.
And not the most optimized way to do it.
Though the workaround can be simply disabled by disabling the line define WINDOWS_EMPTY_LINE_READING_ERROR_FIX. So I really hope Qt will fix this.
In the latest commit I've solved the rendering of the blue-selection square-ranges.
Square at the last line is still incorrect. But that's the has the similar cause as the other last-line bug.
You're bounty remarks sounds fair, no problem.
Btw. I still haven't tested it in on Linux, that would be the next step, after solving the last-line issue.
Idea for last-line workaround is to also add a virtual endline that's always there.
Initial testing with the editor is looking positive 👍
Nice to hear! I hope to fix the 'last line' issue this week. (I'm pretty busy with work now that has piled up since my vacation)
Hi @vadi2, just updated edbee. It fixes the last line reading issue, it looks very good now.
I improved rectangle drawing on selections (though this isn't always 100% correct)
Btw. It seems Windows reader requires the Windows endline symbols of \r\n. I changed ' \n' to '\r\n' (of course)
I removed the QAccessibleEditableTextInterface + methods, the readers never calls them. And I didn't implement/test the windows hack in these methods. So better to remove them.
Further, just now, I found a bug in the undo-operation (on the Mac) .
Undo sends the wrong 'old' text in the onChange operation which can cause an index out of bounds. (at the end of the document)
I'm looking into this now, problem is, I don't seem to have access to the correct old text. (need to figure this out, perhaps do some edbee changes, I need this info in the event!).
Great work!!
Undo and the delete events both require the old text which is a bit tough 🤔
Just fixed the undo-crashing opertion. This was a combination of the wrong old-text being used.
This is fixed by added support to supply on the oldText on a with the TextChange signal.
I also had to add some extra bound checks retrieving texts, because the reader of OS X,
somethems fetches a negative length text.
Regarding testing with NVDA, try pressing insert+S to kill the speech, theoretically, it will not stop it from going into the speech viewer, and you don't have to listen to it.
This is great stuff. Could you also check that it works on Linux with Orca too?
Hi @vadi2 Just been testing on Linux with Orca. It reads (text entry + selections), but not the caret movement.
Trying to figure this out. Very strange
OK. Accesifier is a nice tool to debug this with, it shows events as they happen
@vadi2, where can I find this accesifier debug tool? I can't find it (Tried to google it, tried to find it in the linux packages)
My fault, it is actually Accerciser - https://wiki.gnome.org/action/show/Apps/Accerciser?action=show&redirect=Accerciser
Thanks, installed and now debugging. Getting a bit closer I guess
object:text-caret-moved(1, 0, 0)
source: [text | ]
application: [application | edbee]
screen-reader:mode-changed:caret-tracking(1, 0, =). #<= somehow this isn't happening in edbee (qtextedit does change this mode)
source: [text | ]
application: [application | edbee]
sender: [application | orca]
screen-reader:region-changed(1, 2, 0)
source: [text | ]
application: [application | edbee]
sender: [application | orca]
Are you raising a QAccessibleTextCursorEvent on caret move? This is what we do - https://github.com/Mudlet/Mudlet/blob/development/src/TTextEdit.cpp#L2727-L2732
Thanks for the reply.
Yes I tried the caret event. (I'm a bit struggling when to send it. The cursor event = the selection event)
I tried sending this even explicitly but that didn't change anything.
I saw Mudlet asks the caret position from the QAccessibleTextInterface. Tried that also, but without success.
Sometimes I get the following message (especially when moving the caret)
QSpiApplication::keyEventError "org.freedesktop.DBus.Error.NoReply" "Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken."
I tested with the keyEvent->accept() methods. (didn't do that)
Edbee didn't call the parent method QWidget::keyReleased, also tried to fix net no success.
So I'm still searching for a solution.
I have been reading the caret part of the ia2 accessibility implementation guide https://wiki.linuxfoundation.org/accessibility/ia2/ia2_implementation_guide
I need to try to
- Handle -1 and -2 values. (Didn't implement that)
- and explicitly send a caret event on focus.
Aha, nice find!
After a lot of debugging I found it!
Not reading the caret happend because the textSelectionCount method always returned 1 or more, even if there wasn't a selection. (Technically edbee considers every caret/range a selection (which can be empty). There's always 1).
To make the orca reader read the character at the caret when moving:
textSelectCountshould returns 0 if no selection is available- a
QAccessibleTextCursorEventis fired when no selection is available.
Nice! Sounds like things that could benefit other screen readers too.
I am at a conference this week and only have access to macOS, next week I will be back home and do a final pass with Windows/Linux as well and then we are done.
I've tested it out on Windows, macOS, and Linux - things are working well!
Re: #134 (comment) I'm having trouble finding users to test this - ironically they are not interested - so I'll accept it as-is. Thanks a lot for your work!
Sent me your Paypail details (you can should find my email here), I'll do the transfer.
Good to hear. I you (or other users) find something please let me know. Thanks
(I will email you the details)
Nice!
Now that this is in, it sounds like a good time for a new release :) the last one was a while ago, and a lack of them makes it really difficult for Linux package managers to package the project.