Some commands on the new keyboard emulator do not work
jdcai opened this issue ยท 32 comments
Describe the bug
When I use commands with control + shift
like control + shift + f
it does not work. It is registering that the command has been said but it does not do control + shift + f
. It seems to do something else. When I say control + shift + b
in chrome it seems to do something that causes my screen to flash instead of what the command is supposed to be.
If I say queue + b
though it will toggle the bookmark bar properly in chrome.
Another issue is when I say control + slash
it does not work and gives me this error:
press modifiers plus buttons from button_dictionary_1, non-repeatable
ERROR:action.exec:Execution failed: Keyboard interface cannot type this character: ''
Another issue is that release
does not work for me. I can say hold control
but if I say release control
the command is not recognized.
System:
- OS: Windows 10
- Program: chrome
- Version: 1.6.16 Caster master branch
Thank you for opening up an issue. Let's see what we can do.
Have you enabled the Google Chrome Rule by saying "enable Google Chrome" an "enable Keyboard"?
Notes
The keyboard grammar for keyboard emulation.
https://caster.readthedocs.io/en/latest/readthedocs/Caster_Commands/Keyboard/
Definitions
https://caster.readthedocs.io/en/latest/readthedocs/Rule_Construction/Taxonomy_of_a_Rule/
Spec
is the voice command to trigger an action
Action
is the desired result could be keypress, text, a function and so forth
Command
represents the
spec and action
Desired Commands
-
Action
control + shift + f
, Speccontrol shift foxy
Does nothing in chrome even when manually pressed. Works in other applications such as VS code -
Action
control + shift + b
, Speccontrol shift brov
Successfully toggles chrome favorites bar -
Action
control + slash
Speccontrol slash
I can confirm your issue withcontrol + slash
producesExecution failed: Keyboard interface cannot type this character: ''
@jdcai Let me know if what I've posted above is helpful. if there's any other commands include the keyboard shortcut you expect and the phrase/spec you're using to trigger it.
Another issue is that
release
does not work for me. I can sayhold control
but if I sayrelease control
the command is not recognized.
With my testing below holding/releasing keys works correctly for single modifiers Such as control
but not multiple such as control shift
.
So I test this as well in VS code. If control + F
is pressed the find modal box appears.
hold control
# nothing visually happens as expectedpress foxy
# dialog box appears effectively emulatingcontrol + F
release control
# nothing visually happens as expectedpress foxy
# emulates pressing thef
but does notcontrol + F
. Thereforecontrol
has been released successfully
Now the same procedure control + shift + f
which requires an investigation.
- Highlight some text
1.1hold control shift
# nothing visually happens as expected press foxy
# sidebar find dialog box does not appear as expected.release control shift
shift control foxy
@jdcai can you try the steps above?
@kendonB The output for the keyboard emulation doesn't inform what keys are pressed.
press modifiers plus buttons from button_dictionary_1, non-repeatable
This could be improved and passing the keys into rdescript
I have both rules enabled.
Action control + shift + f
, Spec control shift foxy
Doesn't seem to work in VSCode for find in files. It does work in notepad++ though for find in files.
Action control + shift + n
, Spec control shift novakeen
Doesn't seem to work in Chrome for opening incognito window.
Action control + shift + P
, Spec control shift prime
Works in Firefox for opening private browsing
I realized that I had this transformer which is causing release
to become releft
. How would I change it so that it doesn't affect release
?
<<<SPEC>>>
sauce -> up
dunce -> down
lease -> left
ross -> right
<<<NOT_SPECS>>>
sauce -> up
dunce -> down
lease -> left
ross -> right
Repeating your steps I have the same results by saying releft
instead of release
.
When I try to do hold on multiple like hold control shift
I see this error:
ERROR:action.exec:Execution failed: Cannot use direction with modifiers.
.
Action
control + shift + f
, Speccontrol shift foxy
Doesn't seem to work in VSCode for find in files. It does work in notepad++ though for find in files.
VSCode make sure text is highlighted
Action
control + shift + n
, Speccontrol shift novakeen
Doesn't seem to work in Chrome for opening incognito window.
That is is the correct shortcut However even on my Windows 10 machine control + shift + n
does not seem to open up a new private tab even when pressing keys. is it the same for you? If it doesn't work with the keyboard it definitely will work to dragonfly.
I realized that I had this transformer which is causing
release
to becomereleft
. How would I change it so that it doesn't affectrelease
?
Oh that's an interesting edge case as transformers do replace substrings. That has its pros and cons obviously.
<<<SPEC>>> (for specs only)
<<<EXTRA>>>
<<<DEFAULT>>>
<<<NOT_SPECS>>> (for extras and defaults, but not specs)
<<<ANY>>> (for specs, extras, and defaults).
If you are to keep your transformers <<<NOT_SPECS>>>
that would keep releft
from happening. I would have to review the code to remember how many specs include the directional keys.
I tried it on another computer using Kaldi with debugging messages and looks like it's doing control+windows+shift
instead of control+shift
. I have the same things happening on this computer and on my other computer which uses dragon.
https://superuser.com/questions/1127463/what-does-ctrlwinshiftb-do-in-windows Looks like control + windows + shift + b
restarts the graphic driver which was why I was seeing my screen flash black when saying control + shift + b
.
DEBUG:kaldi.compiler:parse_output(u'#nonterm:rule25 control shift vest #nonterm:end')
Level 12:engine:Alignment (word,time,length): [(u'#nonterm:rule25', 0, 0), (u'control', 2880, 12480), (u'shift', 15360, 10560), (u'vest', 25920, 13440), (u'#nonterm:end', 39360, 0)]
Recognized: control shift vest
DEBUG:action.exec:Executing action: RegisteredAction() ({'button_dictionary_1': 'v', '_grammar': Grammar(g27), '_rule': Keyboard(keyboard), '_node': Node: Alternative(...), [u'control', u'shift', u'vest'], 'modifier': 'cws-'})
press modifiers plus buttons from button_dictionary_1, non-repeatable
DEBUG:action.exec:Executing action: ['%(modifier)s%(button_dictionary_1)s'] ({'button_dictionary_1': 'v', '_grammar': Grammar(g27), '_rule': Keyboard(keyboard), '_node': Node: Alternative(...), [u'control', u'shift', u'vest'], 'modifier': 'cws-'})
DEBUG:action.exec:['%(modifier)s%(button_dictionary_1)s']: Parsing dynamic spec: 'cws-v'
DEBUG:engine:rule execution time 3 ms
DEBUG:engine:dragonfly parse time 0 ms
Level 15:engine:End of phrase: eer=0.00 conf=nan, rule KaldiRule(25, g27::keyboard), u'control shift vest'
Just so you're aware we have other tools as well.
Run_Caster_Test_Engine.bat
It can be run without a speech recognition engine. This is another way that helps differentiate between issue with the code or a misrecognition from the speech recognition backend.
I can't reproduce this using the test engine:
*- Starting Caster v 1.6.16 -*
INFO:command:Calls to mimic() will be delayed by 3.00 seconds as specified
INFO:command:Enter commands to mimic followed by new lines.
control shift char
ERROR:command:Mimic failure for words: control shift char
enable keyboard
INFO:command:Mimic success for words: enable keyboard
control shift char
cs-c
INFO:command:Mimic success for words: control shift char
control shift windows char
cws-c
INFO:command:Mimic success for words: control shift windows char
I changed the rdescript as @LexiconCode suggested
Also can't reproduce using Kaldi on Windows 10 x64 with Python38.
@jdcai can you try with a fresh user folder by editing Run_Caster_Kaldi to something like this?
@echo off
echo Running Kaldi from Dragonfly CLI
set currentpath=%~dp0
echo %currentpath%
echo Add psutil to requirements.txt
TITLE Caster: Status Window
set CASTER_USER_DIR=C:\test_caster_user
C:\Python38\python.exe -m dragonfly load _*.py --engine kaldi --no-recobs-messages
pause 1
I can't reproduce this using the test engine:
I can't reproduce this either. Looking at the debugging output that he provided despite the phrase control shift vest
It's action is indeed being recognized as 'cws-'.
DEBUG:action.exec:Executing action: ['%(modifier)s%(button_dictionary_1)s'] ({'button_dictionary_1': 'v', '_grammar': Grammar(g27), '_rule': Keyboard(keyboard), '_node': Node: Alternative(...), [u'control', u'shift', u'vest'], 'modifier': 'cws-'})
DEBUG:action.exec:['%(modifier)s%(button_dictionary_1)s']: Parsing dynamic spec: 'cws-v'
Test live dragonfly grammars
- extract testme.zip to
Documents\Caster
(testme.py
should be in the same directory as_caster.py
) - open command prompt
Documents\Caster
python -m dragonfly load testme.py --engine kaldi
(this will load onlytestme.py
and not the rest of caster).testme.py
is built with only dragonfly for testing purposes.- Test out the following Specs and see if they register correctly in the debug output.
- control shift brov
- control windows shift brov
- test control windows shift brov
- test control shift brov
- close out the command prompt Window
Test text engine dragonfly grammars
Text Engine Instructions
The text engine does not rely in any speech recognition engine and helps us differentiate issues with code versus issues with the speech recognition backend.
Type commands to emulate as if they are being dictated by voice.
lowercase mimicscommands
, UPPERCASE mimicsfree dictation
Upper and lowercase words can be mixed e.gsay THIS IS A TEST
Edit the--delay 3
to change command delay in seconds.
The delay allows user to switch to the relevant application to test commands
- Reopen command prompt and run the text engine
python -m dragonfly test testme.py -l DEBUG --delay 3
- type the following one phrase at a time:
- control shift brov
- control windows shift brov
- test control windows shift brov
- test control shift brov
Test text engine caster grammars
- Backup and delete your castor user directory.
- Reopen command prompt and run the text engine
python -m dragonfly test _caster.py -l DEBUG --delay 3
- type the following one phrase at a time:
- control shift brov
- control windows shift brov
Depending on the results above would you be willing to connect over gitter and perhaps we can arrange a time for me to see remotely what this looks like first-hand?
@kendonB However Spec control slash
is a different bug that is reproducible.
Execution failed: Keyboard interface cannot type this character: ''
I tried using a different user folder as @kendonB suggested using your Run_Caster_Kaldi.bat code and I still have the cws problem.
@LexiconCode I ran your commands using kendonB's code by changing the python line to your lines. I'm using Python 2.7
@echo off
echo Running Kaldi from Dragonfly CLI
set currentpath=%~dp0
echo %currentpath%
echo Add psutil to requirements.txt
TITLE Caster: Status Window
set CASTER_USER_DIR=C:\test_caster_user
python -m dragonfly test _caster.py -l DEBUG --delay 3
pause 1
Test Live Dragonfly Grammars
The specs work as expected.
Test text engine dragonfly grammars
The specs work as expected.
Test text engine caster grammars
control shift brov still does cws-b instead of cs-b. Debug message below:
DEBUG:grammar.decode: rollback: Choice(..., name='button_dictionary_1')
DEBUG:grammar.decode: attempt: Compound(u'brov')
DEBUG:grammar.decode: attempt: Literal([u'brov'])
DEBUG:grammar.decode: success: Literal([u'brov'])
DEBUG:grammar.decode: -- Decoding State: 'control shift brov >> '
DEBUG:grammar.decode: success: Compound(u'brov')
DEBUG:grammar.decode: success: Choice(..., name='button_dictionary_1')
DEBUG:grammar.decode: success: Sequence(...)
DEBUG:grammar.decode: success: Compound(u'<modifier> <button_dictionary_1>')
DEBUG:grammar.decode: success: Alternative(...)
DEBUG:grammar.decode: success: Keyboard(keyboard)
DEBUG:action.exec:Executing action: RegisteredAction() ({'button_dictionary_1': 'b', '_grammar': Grammar(g4), '_rule': Keyboard(keyboard), '_node': Node: Alternative(...), [u'control', u'shift', u'brov'], 'modifier': 'cws-'})
press modifiers plus buttons from button_dictionary_1, non-repeatable
DEBUG:action.exec:Executing action: ['%(modifier)s%(button_dictionary_1)s'] ({'button_dictionary_1': 'b', '_grammar': Grammar(g4), '_rule': Keyboard(keyboard), '_node': Node: Alternative(...), [u'control', u'shift', u'brov'], 'modifier': 'cws-'})
DEBUG:action.exec:['%(modifier)s%(button_dictionary_1)s']: Parsing dynamic spec: 'cws-b'
INFO:command:Mimic success for words: control shift brov
@jdcai @LexiconCode I can reproduce with "control shift brov"
So it's something to do with which spec. This is on the text engine
So it's something to do with which spec. This is on the text engine
Hmm I agree. I believe both of us as far tested under Python 3 time to revisit with 2.
I do not think c-shift
is valid. spec control shift
DEBUG:action.exec:Executing action: ['%(modifier)s%(button_dictionary_1)s'] ({'button_dictionary_1': 'shift', '_grammar': Grammar(g4), '_rule': Keyboard(keyboard), '_node': Node: Alternative(...), [u'control', u'shift'], 'modifier': 'c-'})
DEBUG:action.exec:['%(modifier)s%(button_dictionary_1)s']: Parsing dynamic spec: 'c-shift'
INFO:command:Mimic success for words: control shift
So it's something to do with which spec. This is on the text engine
My time will be limited over the next 3 1/2 days as I have overnight shifts. At least we have a start.
@kendonB when does the following work for you and does it emulate the same behavior as your original code?
I haven't figured out why this works yet in my limited testing due to time constraints.
def cat_spec_and_reverse(s1, s2):
return ("(" + s1 + " " + s2 + ")" + " | " + "(" + s2 + " " + s1 + ")")
def cat_spec_and_reverse_3(s1, s2, s3):
action_cat_spec_and_reverse = cat_spec_and_reverse(s1, s2)
return (action_cat_spec_and_reverse + s3 + " | (" + s1 + " " + s3 + " " + s2 + ")" + " | (" + s2 + " " + s3 + " " + s1 + ")" + " | (" + s3 + " " + s2 + " " + s1 + ")")
It doesn't emulate the behaviour of the original code - but it may correct a bug in the original code! It also works for me (for a few examples at least) making that change - I will still have to check it properly when I'm at my linux computer (python debugging is easier there). I'll be offline for 3 days then will look. Feel free to push this fix if you find it addresses the bugs.
I saw the control shift char bug both with and without the function change. It may be intermittent...
The pull request fixes the issue with cws.
There are also two other issues found on the keyboard commands.
control + slash
gives this error:
press modifiers plus buttons from button_dictionary_1, non-repeatable
ERROR:action.exec:Execution failed: Keyboard interface cannot type this character: ''
hold control shift
gives this error:
ERROR:action.exec:Execution failed: Cannot use direction with modifiers..
The pull request fixes the issue with cws.
There are also two other issues found on the keyboard commands.
control + slash
gives this error:press modifiers plus buttons from button_dictionary_1, non-repeatable ERROR:action.exec:Execution failed: Keyboard interface cannot type this character: ''
hold control shift
gives this error:
ERROR:action.exec:Execution failed: Cannot use direction with modifiers..
is that with this PR or without it?
is that with this PR or without it?
Happens with and without the PR
is that with this PR or without it?
Happens with and without the PR
@jdcai This should be fixed in the latest master.
Oh I only had the pull request code. I updated to master and control + slash works now. hold control shift command was removed?
Awesome.
hold control shift command was removed?
That should not have been removed.
hold control shift command was removed?
Sigh no it's not working as intended. #877
I didn't think that I had ever implemented "hold control shift" @jdcai did that command really work as you expected before? The current code I believe only works for single keys
What's the use case for "hold control shift"?
it should work now with "hold control hold shift" btw
I'd prefer not to allow for multiple keys with hold and release. That's just more scope for misrecognition with very limited use case
I'd prefer not to allow for multiple keys with hold and release. That's just more scope for misrecognition with very limited use case
That makes sense, one thing I would like to do in a future PR is register the keys that are held down. that way they can be an generic release held keys
without specifying the individual keypresses.
What's the use case for "hold control shift"?
I could see use case for this when wanting to perform multiple command keypresses held. For instance if you hold "hold control shift" followed by directional keys you can say select text most programs. Obviously dedicated commands are better for this but for developing or troubleshooting these would be nice. Most importantly it provides consistency with dictating commands.
If I can find a convenient way and to add -
Symbols when they're used as a modifier key this should be pretty simple to implement. Does that sound okay?
Most importantly it provides consistency with dictating commands.
Not sure what you're referring to here?
If I can find a convenient way and to add - Symbols when they're used as a modifier key this should be pretty simple to implement.
Also don't understand this sorry