Visual cues experimentation for Augmented Reality Obstacle Avoidance
This project establishes a visual cue system for the HoloLens 2, as used in the paper "Using augmented reality to cue obstacles for people with low vision." It uses Unity v2019.4.40f1 and the Windows Mixed Reality XR plug-in.
The core functions of the project are:
- Course Alignment - Align with a predetermined physical obstacle course layout using QR codes. (QR codes created here.) This course is set up over a 1.8m x 15m portion of hallway and can take one of 8 layouts.
- Cue Display - Display visual cues in one of three modes: World-Locked (aka Collocated), Heads-Up (aka HUD), or Combined (both other cue types at once).
- Performance Logging - Log the participant's position and rotation over time as they walk the obstacle course.
Note that the obstacles used are:
- Low (a 2” x 2” dark gray bar of foam spanning the hallway that participants stepped over)
- High (a ~2” wide white streamer hung horizontally at eye level that participants ducked under)
- Wide (a 2m tall x 1.5m wide folding room divider that participants walked around)
The QR code printout should be placed to the left of the start of the course and positions fine-tuned using voice commands or keyboard shortcuts.
The following secondary functions support the core functions.
- Calibration Mode - displays all four sides of the heads-up/HUD cues and removes distance limitations on world-locked/collocated cues. Helpful for ensuring mode and obstacle placements are correct.
- HUD Calibration - adjust the Heads-Up/HUD cues left or right and condense them for users with vision loss in one eye.
- Debug Text - display a floating window with debugging information to facilitate setup.
- Voice Commands - control the experience using voice commands or keyboard input from a bluetooth keyboard.
These functions were created during the prototyping phase but were not used for the main experiment.
- Interface Menu - a floating menu used to control the obstacles and mode.
- Manual Obstacle Positioning - the ability to adjust the size and placement of obstacles relative to one another manually, and to save a particular layout and reset to it. (Note that saving is within-session only - if the application restarts, saved positions are lost.)
- Show/Hide Obstacles - functionality operated from the interface menu to show or hide additional obstacles one at a time in order to allow manual placement of multiple obstacles.
These can be found in the latest scene (currently Visual Cues v7b_Hallway).
-
Mixed Reality Toolkit - the core component enabling HoloLens functionality. Read more at Microsoft's GitHub. Note that speech commands and their associated keyboard shortcuts are entered here under Input > Speech > Speech Commands.
-
Main Camera - The camera represents the HoloLens. Note that the Clipping Plane and Culling Mask fields are adjusted to affect world-locked cue visibility.
-
HUD Revised - This holds the HUD_Revised_v2 script, which controls the heads-up cues.
-
Text To Speech Object - This holds the Text To Speech script and an audio source, used to announce updates to user.
-
HUD Frame - This holds the HUD Cue North/East/South/West objects, which are what enable or disable to show heads-up cues.
-
Debug Canvas - This shows the debug text while enabled.
-
-
-
Obstacle Manager - This holds three key scripts: Obstacle Manager, Experiment Logger, and Speech Input Handler. Note that any voice commands/keyboard shortcuts used in the MRTK must be matched to commands in Speech Input Handler.
-
QR Codes Manager - This holds two of the necessary scripts for QR code usage: QR Codes Manager and QR Codes_AROA.
-
Collocated Cues - This holds the set of six potential obstacles, as well as a calibration obstacle used to help ensure placement is correct. Each obstacle has an Object Manipulator, Bounds Control, and Constraint Manager script attached to enable manual manipulation, though this is locked by default.
-
Interface - This is the interface menu. It uses the Billboard, SolverHandler, RadialView, Constraint Manager, and Object Manipulator scripts from the MRTK to allow automatic and manual placement of the menu.
-
Other Objects - These are objects representing the floor and Calibration QR Code to help place obstacles. They are kept disabled during use.
The process of building the unity project and deploying to HoloLens can be very delicate and sensitive to error. Here is what worked for us.
- In Unity, go to File > Build Settings.
- Under "Scenes In Build," make sure the most recent one (currently Visual Cues v7b_Hallway) is the only one checked.
- Set platform to Universal Windows Platform.
- Set Target Device to HoloLens.
- Set Architecture to ARM64.
- Set Build Type to D3D Project.
- Press Build. Choose the Builds folder of Visual Cues as the target destination.
- Connect the HoloLens to your PC via a USB-C cable.
- After the project builds, open the solution in Visual Studio. (Make sure you have the necessary Visual Studio extensions installed.)
- Adjust Configuration to Debug.
- Adjust Platform to ARM64.
- Adjust Target to Device.
- Click "Run Without Debugging."
- Once you hear the bootup sound on the HoloLens, it is safe to disconnect the USB-C cable.
If this doesn't work, I suggest asking on the HoloDevelopers Slack.
Prototype setup and conduct as used in the paper:
- Set up an obstacle course according to one of the predetermined layouts. QR Codes should go to the left of the starting line at approximately 1.4m. (May need to adjust so wide and low obstacles are correctly placed on the floor.) High obstacles should go at head height for the participant.
- Make sure a computer is synced to the HoloLens 2. (See "Remote Operation" below.)
- Fit HoloLens 2 to participant. Run eye calibration. (A bluetooth keyboard is helpful for this.)
- Launch AROA app. (This can be done via the Windows Device Portal or by having the participant say "Obstacle Avoidance" while the Start menu is open.)
- Use the up or down arrows to adjust high obstacles to eye level for the participant.
- If the participant has vision loss in one eye, adjust the heads-up cues using the Shift HUD Left/Right and Increase/Decrease HUD Size commands.
- Ask participant to scan QR code for the current course.
- Ask participant to look forward and close their eyes so they don't see the whole course preview ahead of time.
- Using Toggle Calibration, Rotate Left, and Rotate Right, adjust the world-locked visual cues to match the real world position of obstacles. You may need to ask the participant to re-scan the QR code.
- Turn off calibration mode and ensure that the app is set to the correct cue mode. The debug window will show your mode, layout, and direction, and will display "OK TO EXPERIMENT" if calibration mode is off. Make sure to turn off the debug text using the keyboard command 9 before starting.
- Have the participant stand at the center of the starting line. Count them off, and press 3 for Begin Logging as they start. When they cross the finish line, press 4 for End Logging. (Note that this will automatically cause direction to shift from Forward to Backward or vice versa for logging and debug purposes.)
- Repeat the previous step, but starting at the end of the course and going backwards.
- Adjust the layout and repeat steps 7-12 for each additional trial.
- Make sure the computer and HoloLens are on the same wifi network.
- Get the HoloLens’ IPv4 address from Settings/Network and Internet/Properties, e.g. 192.168.1.70
- In a web browser on the laptop, go to http://[IPv4 address] You should see a “Your connection isn’t private” message. Click “Advanced,” and then “Continue to…” It will ask you for the username and password you used to connect your HoloLens.
- You should now see the Windows Device Portal.
To view a stream of what the HoloLens sees:
- Go to Views / Mixed Reality Capture
- Under “Capture,” unselect everything except Holograms and PV Camera, and set preview quality to Low
- Hit “Live Preview.” (Note that there is several seconds of lag, and this may create an additional resource strain on the HoloLens. You may want to disable this preview during experimentation to improve HoloLens performance.)
To launch an application:
- Select it from the dropdown list under “Installed Apps.” (The AROA app is labeled “Obstacle Avoidance”.)
- Hit Start.
To force quit an application:
- Go to Views / Apps
- Under Running Apps, press the X next to the app you want to quit.
- Confirm close on the popup.
To run eye calibration:
- Launch the Settings app.
- Under System / Virtual Input (or using a bluetooth keyboard), type in “Run eye calibration” and hit enter.
- You should be able to use Tab and Enter keys to get to the Calibration page, and then select Run Eye Calibration. (It should be the first item selected, so just press enter again.) Note that you can use shift+enter to move focus backwards.
To retrieve logged data:
- Ensure you are using http:// instead of https:// in your web browser.
- Ensure “SSL Required” is off under Settings/System/Preferences on the browser.
- Navigate to System/File Explorer/LocalAppData/Obstacle Avoidance (AROA_…)/LocalState.
- Click the Save icon next to each file.
Voice commands will rarely recognize anyone but the person wearing the HoloLens. Keyboard shortcuts can be used after connecting a bluetooth keyboard to the HoloLens. (We used the Logitech K380 Multi-Device Bluetooth Keyboard.)
| Command | Key | Description |
|---|---|---|
| Toggle Collocated Cues | 1 | Turns visibility of world-locked/collocated cues on or off. Default is on. |
| Toggle HUD Cues | 2 | Turns visibility of heads-up/HUD cues on or off. Default is on. |
| Begin Logging | 3 | Begins logging an obstacle course trial. |
| End Logging | 4 | Stops logging an obstacle course trial and saves record. |
| Save Position | 5 | Saves relative positions of obstacles. [DEPRECATED] |
| Reset Position | 6 | Restores relative positions of obstacles to last saved position. [DEPRECATED] |
| Toggle Interface | 7 | Toggles visibility of interface. Default is off. |
| Obstacles Front | 8 | Resets obstacles to be in front of user. |
| Toggle Debug | 9 | Toggles visibility of debug text. Default is on. |
| Shift HUD Right | L | Shifts heads-up cues to the right. |
| Shift HUD Left | J | Shifts heads-up cues to the left. |
| Increase HUD Size | I | Expands heads-up cues horizontally. |
| Decrease HUD Size | K | Contracts heads-up cues horizontally. |
| Toggle Calibration | C | Removes distance limit on world-locked obstacles and displays calibration obstacle; sets all heads-up cues to On. Default is on. |
| Toggle Gesture Lock | G | Toggles ability to manipulate obstacles manually. Default is off. |
| Increase Front Angle | [ | Increases the size of the "Front Angle", enabling heads-up cues to point to obstacles further to the side of the user. Default is 75 degrees with every button press adding 5 degrees, to a maximum of 90 degrees. |
| Decrease Front Angle | ] | As above, but decreases by 5 degrees to a minimum of 0 degrees. |
| Increase HUD Threshold | ; | Increases the HUD threshold. The higher the number, the more the user will have to look away from the obstacle to activate heads-up cues. Default is 0.15 with every button press adding 0.05 degrees to a maximum of 1. |
| Decrease HUD Threshold | ' (apostrophe) | As below, but decreases by 0.05 to a minimum of 0. |
| Rotate Left | <- (left arrow) | Rotates obstacles to the left by 1 degree. |
| Rotate Right | -> (right arrow) | Rotates obstacles to the right by 1 degree. |
| Raise High Obstacles | up arrow | Raises high obstacles by 0.0254m (1 inch). Default height is 60 inches. |
| Lower High Obstacles | down arrow | Lowers high obstacles by 0.0254m (1 inch). Default height is 60 inches. |