/Labs-GazeControls

Set of user controls that can be reused in different applications with eye gaze input.

Primary LanguageC#MIT LicenseMIT

Gaze Controls

Eye gaze as a form of input is similar to mouse, pen and touch in the sense that it provides a stream of coordinates that the user is looking at. But when it is actually used for input and interaction in applications, the design considerations are significantly different.

Eye gaze is less precise than pen, touch and mouse. The measurement errors are larger due to lighting and environmental conditions. There is a larger variance among user populations. There is a diversity in the quality and calibration accuracy among different trackers. When these factors are considered, it becomes apparent that a user interface designed for mouse, pen or touch cannot be directly used for effective eye gaze interaction. Just as user interfaces originally designed for mouse were modified to suit touch interaction, we need to modify the existing user interfaces to be compatible with eye gaze input.

The GazeControls library, built on the GazeInteraction Library, includes a set of user controls that can be reused in different applications with eye gaze input. Instead of trying to design controls for all forms of input simultaneously these set of controls are designed primarily for eye gaze input.

Prerequisites

Since the GazeControls library is built on top of the GazeInteraction library, the first set of prerequisites are the same as the prerequisites for the GazeInteraction library.

There are additional prerequisites for each of the controls and they are listed below in the context of the documentation for the specific control.

Nuget Feed

To use the GazeControls library, please do the following:

  1. Add a new nuget feed either in nuget.config or Visual Studio by clicking on Tools...Options...Nuget Package Manager...Package Sources and adding a new entry with the following details:
    • Name: CommunityToolkit-Labs
    • Source: https://pkgs.dev.azure.com/dotnet/CommunityToolkit/_packaging/CommunityToolkit-Labs/nuget/v3/index.json
  2. Add a reference to CommunityToolkit.Labs.Uwp.GazeControls from the above nuget feed

Please refer to the documentation below on how to use specific controls.

Supported controls

The library currently supports the following user controls.

  • GazeKeyboard
  • GazeFileOpenPicker
  • GazeFileSavePicker
  • GazeScrollbar

GazeKeyboard

One of the most common uses of eye gaze input is by users with mobility impairments. And the most common task in such scenarios is text input, especially if the user also has a speech impairment, like in the case of users with ALS. Text input with eye gaze is particularly challenging for an assorted set of users. One aspect of that challenge is that it is impossible to design an optimal keyboard layout that works for all users in all occasions. The GazeKeyboard control in this library is intended to address this problem by making it easy to define new keyboard layouts and include them in your gaze application.

GazeKeyboard Prerequsites

The GazeKeyboard control provides a way to define custom keyboard layouts (including styling) and injects the specific key the user dwells on into the application. To perform the key injection, it relies on the inputInjectionCapability. Please make sure to add this capability to your application's Package.appxmanifest as follows:

<Capabilities>
   <DeviceCapability Name="inputInjectionCapability" />
</Capabilities>

The application should also add a reference to the GazeInteraction library nuget.

Builtin layouts

The GazeKeyboard control comes built in with three layouts.

  • MinAAC. This layout defines the minimal possible English layout to enable an AAC (Augmentative and Assistive Communication) application. It simply contains English alphabet and a few editing keys.
  • FullKeyboard. This layout supports all the keys for a full hardware keyboard and also some features found on a software only keyboard, like emojis. It short it is an example of all the features supported by the GazeKeyboard control.
  • TwoStageKeyboard. This unique feature shows off a way to still support text entry even when the gaze accuracy is extraordinarily low. It takes a minimum of two button presses to enter one alphabet. But it shows how to define a custom layout and support text entry as long as the user can reliably gaze on a four by four grid of buttons on the screen.

(In the source code, you will see a fourth keyboard layout named FilenameEntry. This layout is for use by the GazeFilePicker dialog.)

To use any of the above layouts, please do the following:

  • Add the library namespace to the XAML page as follows:

    <Page
  ...
  xmlns:gc="using:Microsoft.Toolkit.Uwp.Input.GazeControls"
  ...
  >

  • Reserve a space in your layout for the gaze keyboard and instantiate the keyboard there.

      <Grid>
      <Grid.RowDefinitions>
          <RowDefinition />
          <RowDefinition />
      </Grid.RowDefinitions>
  <gc:GazeKeyboard x:Name="GazeKeyboard" Grid.Row="1" />

  </Grid>

  • Load a keyboard layout after the Page has loaded with code similar to the following:

    private async void LoadKeyboardLayout()
{
    var uri = new Uri($"ms-appx:///CommunityToolkit.Labs.Uwp.GazeControls/KeyboardLayouts/MinAAC.xml");
    var layoutFile = await StorageFile.GetFileFromApplicationUriAsync(uri);
    await GazeKeyboard.TryLoadLayoutAsync(layoutFile);
    GazeKeyboard.Target = TheTextBox;
    GazeKeyboard.PredictionTargets = new Button[] { Prediction0, Prediction1, Prediction2 };
}

Custom layouts - Simple

You can also define your own custom layouts, include it in your application and change the layouts on the fly by assigning the LayoutUri property of the keyboard control to the URI of your custom keyboard layout as shown above.

Custom layouts are specified directly using XAML and a few attached properties that dictate their behavior. The best way to create new layouts, is to use one of the built-in layouts as a starting point and copy and modify them to your own needs.

NOTE: Keyboard layouts are specified in XAML but the file is saved with a .xml extension

The behavior of the button when it is clicked is governed by a few rules:

  • The top-level element must be a Grid. (In the simple case, having a name for the Grid is optional.)

  • All the styling for the buttons must be contained within the same XAML file in one of the Resources sections.

  • All the layout elements must be subclasses of ButtonBase.

  • If a button only has the Content property defined, then the content string is injected into the application. E.g., if you have a button defined as <Button Content="p" /> the p key injected when the button is pressed. (This example has been shortened for brevity. In principle, you can add any other Button related property like Grid.Row, Style etc.)

  • If a button also has a VK property defined, then the Content property only determines the appearance of the button. The integer value of the VK property is injected when the button is pressed. In the example below, a backspace key is injected when the button is pressed.

    <Button Style="{StaticResource Symbol}" Content="&#xE750;" k:GazeKeyboard.VK="8"/>

  • If a button has the VKList property, the control expects a set of virtual key codes as Int32 values and injects that list of keys in sequence. E.g. in MinAAC.xml you can see the VKList property assigned for clearing the textbox as follows.

      <Button Grid.Row="2" Grid.Column="9" Style="{StaticResource Symbol}" Content="&#xE74D;">
      <k:GazeKeyboard.VKList>
          <x:Int32>17</x:Int32> <!-- VK_CONTROL -->
          <x:Int32>65</x:Int32> <!-- VK_A -->
          <x:Int32>8</x:Int32>  <!-- VK_BACK -->
          <x:Int32>8</x:Int32>  <!-- VK_BACK -->
          <x:Int32>65</x:Int32> <!-- VK_A -->
          <x:Int32>17</x:Int32> <!-- VK_CONTROL -->
      </k:GazeKeyboard.VKList>
  </Button>

    In the above example, when this button is pressed, it injects the following sequence of keys:

    • Control key down
    • A key down
    • Backspace key down
    • Backspace key up
    • A key up
    • Control key up

    As you can see, if a key occurs twice in the list, the first is interpreted as a down key, and the second as an up key.

Custom layouts - Advanced

When the number of keys in the layout is larger than what the application can display (and still have space left over for the actual application needs), it can choose to split up the layout into multiple pages. When this is done, the XAML layout for the custom layout should follow the rules below:

  • The top level Grid should have a name specified with x:Name property.

  • There should be a GazeKeyboard.PageList in the layout. This node should include a list of strings that specify the names of other Grid notes, which define the separate pages in the layout. The example below is taken from FullKeyboard.xml

      <k:GazeKeyboard.PageList>
      <x:String>MainPage</x:String>
      <x:String>UppercasePage</x:String>
      <x:String>NumbersPage</x:String>
      <x:String>EmojiPage</x:String>
  </k:GazeKeyboard.PageList>

    The layout parser now expects to find four different Grid nodes with the names of MainPage, UppercasePage, NumbersPage and EmojiPage.

  • Only the main layout grid should have its Visibility property set to Visible and all the other pages should have it set to Collapsed.

  • Changing pages. In order to load a new layout when a particular button on the current layout is pressed, the button should include the GazeKeyboard.NewPage property and set the value to the name of one of pages specified in the PageList property. In the following example, keyboard layout changes to the NumbersPage when the button is pressed.

    <Button Content="123" k:GazeKeyboard.PageContainer="FullKeyboard" k:GazeKeyboard.NewPage="NumbersPage"/>

  • TemporaryPages. The GazeKeyboard's layout engine supports the notion of temporary pages. A temporary page takes over the layout when a button defines GazeKeyboard.TemporaryPage property and sets the value to one of the pages found the in the PageList property. When any button is pressed even once on the temporary page, the visible page layout immediately transitions back to the previous layout. This mechanism is used to support changing the keyboard layout to upper case when the Shift key is pressed and to also support two stage keyboard layouts. When a button defines a GazeKeyboard.TemporaryPage property it also needs to define GazeKeyboard.PageContainer property to inform the layout engine as to which page layout to load back. The example below, illustrates this.

    <Button Style="{StaticResource Symbol}" Content="&#xE752;" k:GazeKeyboard.PageContainer="FullKeyboard" k:GazeKeyboard.TemporaryPage="UppercasePage" />

  • Unicode keys. If a unicode character is to be injected, the button should specify the GazeKeyboard.Unicode property and set its value to the Unicode character to inject as shown in the example below.

    <Button Style="{StaticResource Alpha}" Content="&#x1f600;" k:GazeKeyboard.Unicode="&#x1f600;"/>

GazeKeyboard Properties

Property Type Description
Target TextBox Gets or sets the target text box for injecting keys
PredictionLanguage string Gets or sets the text prediction language
PredictionTargets Button[] Gets or sets the prediction targets buttons. When text prediction is available, the content of the buttons it set to the prediction text.

GazeFileOpenPicker and GazeFileSavePicker

The file picker controls provides a gaze optimized subset of the features of the full OS native file picker dialog. Since this control needs to enumerate the file system, appropriate capabilities need to be declared in the Package.appxmanifest file in the application that uses this control. E.g. if an application is going to access the documents folders, then the application needs to add the following line to the <Capabilities> sectioni of Package.appxmanifest.

<Capabilities>
  <DeviceCapability Name="documentsLibrary"/>
</Capabilities>

Features

Both dialogs support a common set of features:

  • Enumerate files and folders in the directories the application has permission to access and display them in a gaze friendly large icon view.
  • Ability to navigate folders using gaze or gaze + switch.
  • Ability to set the file filter type to show only the files matching a specific set of extensions.
  • Ability to define a set of folders as "favorite" starting points for the application
  • A gaze friendly scroll bar.

GazeFileOpenPicker

GazeFileOpenPicker supports the selection and opening of existing files and does not support creation of new folders or files.

GazeFileSavePicker

GazeFileSavePicker supports the following additional capabilities:

  • Creation of new folders
  • Ability to choose a new filename in a keyboard layout that is optimized for entering file names with gaze
  • An additional warning dialog is displayed when an existing file is about to be overwritten.

Sample Code

The following sample code illustrates how to use the GazeFilePicker dialog for file-open and file-save operations.


private async void OnFileOpen(object sender, RoutedEventArgs e)
{
    var picker = new GazeFileOpenPicker();
    var library = await StorageLibrary.GetLibraryAsync(KnownLibraryId.Documents);
    picker.FileTypeFilter.Add(".txt");
    picker.CurrentFolder = library.SaveFolder;
    await picker.ShowAsync();
    if (picker.SelectedItem != null)
    {
        _textControl.Text = await FileIO.ReadTextAsync(picker.SelectedItem);
    }
}

private async void OnFileSave(object sender, RoutedEventArgs e)
{
    var library = await StorageLibrary.GetLibraryAsync(KnownLibraryId.Documents);
    var picker = new GazeFileSavePicker();
    picker.FileTypeFilter.Add(".txt");
    picker.CurrentFolder = library.SaveFolder;
    await picker.ShowAsync();
    if (picker.SelectedItem != null)
    {
        await FileIO.WriteTextAsync(picker.SelectedItem, _textControl.Text);
    }
}

GazeFilePicker Properties

Property Type Description
CurrentFolder StorageFolder Gets or sets the current folder for the file picker dialog
Favorites List<StorageFolder> Gets or sets the list of storage folders that appear as shortcuts on top of the folder view
FileTypeFilter List<string> Gets or sets the collection of file types that the file open picker displays.
SaveMode bool Gets or sets a value indicating whether this is FileSave dialog or a FileOpen dialog
SelectedItem StorageFile Gets the currently selected file in the dialog as a StorageFile

GazeScrollbar

The builtin scrollbar control that is attached to ScrollViewers is difficult to use with eye gaze because the scroll buttons are too small and not easily targetable with eye gaze. In theory it is possible to re-template the built-in scrollbar and make the scroll buttons and the whole scroll bar larger. But these scroll buttons are not returned as valid elements from hit testing and hence these buttons are not accessible via eye gaze. Therefore, this library supports the functionality of a scroll bar as a standalone control that can be attached to any ScrollViewer.

Usage

In order to use the GazeScrollbar please make the following changes to your code.

Add the GazeScrollbar to your XAML page

The first step is to the add GazeScrollbar control to your XAML page as follows.

<gc:GazeScrollbar Grid.Row="0" Grid.Column="1" x:Name="CurrentFolderScrollbar" Orientation="Vertical" HorizontalAlignment="Stretch" VerticalAlignment="Stretch"/>

The important property above is Orientation which determines whether it is a horizontal or vertical scrollbar. While the scrollbar can be placed anywhere on the page independent of the attached scrollview, it is best positioned in a grid next to the scrollview it is controlling.

Attach the ScrollViewer to GazeScrollbar

Next the scrollbar needs to be attached to the ScrollViewer as follows in a relevant event handler after the visual tree has been loaded.

GazeScrollbar.Attach(scrollViewer);

NB: In some cases the scroll viewer is part of a larger control template and is not directly accessible, e.g. GridView. In such cases, you can access the embedded ScrollViewer by subclassing the GridView (or similar control), overriding OnApplyTemplate and extracting the templated ScrollViewer by calling GetTemplateChild. You can follow the example in the ScrollGridView class in the GazeControls library

Set the line height for the GazeScrollbar

The GazeScrollbar has two pairs of buttons for each direction of scrolling. E.g. to scroll down, it has a button for scrolling one line at a time, and another for scrolling a page at a time. The page size is automatically determined by the size of the viewport and the size of the content in the ScrollViewer. However, the size of the line is scenario dependent. E.g. in the case of text it is the height of the line and in the case of a list view, it is dependent on the height of each item. The GazeScrollbar supports LineHeight and LineWidth properties that can be used to control the distance to scroll when one of the line scrolling buttons are pressed.

GazeScrollbar properties

Property Type Description
Orientation Orientation Gets or sets the orientation of the scrollbar
LineWidth double Gets or sets the distance to scroll horizontally when a line-left or line-right button is pressed
LineHeight double Gets or sets the distance to scroll vertically when a line-up or line-down button is pressed

🧪 This project is under Community Toolkit Labs. What does that mean?

Community Toolkit Labs is a place for rapidly prototyping ideas and gathering community feedback. It is an incubation space where the developer community can come together to work on new ideas before thinking about final quality gates and ship cycles. Developers can focus on the scenarios and usage of their features before finalizing docs, samples, and tests required to ship a complete idea within the Toolkit.