VRySmart
The VRySmart framework aims to integrate smart devices into virtual environments.
The project contains assets for both Windows and Android target platforms. The core of the framework is located under the Assets/Framework folder. The project integrates many different modules and at this point serves as a proof-of-concept or a starting point for further development.
For more details, please check out the CHI 2022 Late-Breaking Work below.
Link to the paper: https://doi.org/10.1145/3491101.3519717
Link to the presentation video: https://www.youtube.com/watch?v=qPkUWzpyVe8
Requirements
The framework is built in Unity using SteamVR and SRWorks for skin segmentation.
Minimum requirements:
- (Important!) The framework relies on the OpenCVForUnity plugin for tracking Aruco markers. You can either purchase this asset here, or use any other alternative for marker tracking.
- Unity 2019.4
- SteamVR
- SRWorks runtime
Hardware:
- HTC Vive Pro
- Smartphone (tested with Pixel 3)
Packages:
- XR Plugin Management
- XR Interactin Toolkit
- Multiplayer HLAPI
- OpenVR (Desktop)
- Input System
- ViveSR
Android support
Packages:
- AR Foundation
- ARCore Extensions
- ARCore XR Plugin
Project settings
Player -> Other Settings -> Active input handling -> Both
Usage
Main scene for the desktop app is: Desktop(OpenCVForUnity+RS+TS).unity
Main scene for the mobile app is: Mobile(RS+TS).unity
To test the scenarios:
- Make sure you added OpenCVForUnity into the assets
- Deploy the mobile app to your smartphone
- If you want to use Camera streaming start webserver.exe - Shortcut from the root folder
- Start Windows app
- Start Android app
- Android app requests host-ip address. Provide the ip-address of the PC where the main VR-app is running and press connect
Scenarios
Currently, four different scenarios are implemented.
Troubleshooting
Mobile app doesn't connect to the VR-host
Check firewall settings on the PC, it might block the connection
Checkout Issues
Please ensure that you have activated git LFS to checkout this repo. If you receive a smudge error, please refer here: Smudge Error while downloading repo
Camera stream is white
Check Firewall settings, consider disabling it for the private network
Compliation conflicts for Android if deprecated scripts ViveSR_StaticColliderPool.cs, ViveSR_StaticColliderInfo.cs from ViveSR package are part of the project
Remove the scripts
Paper
The VRySmart framework originated from a Masters thesis at the Saarland University by Akhmajon Makhsadov. This project was published at the ACM CHI'22 conference as a Late-Breaking Work. DOI: https://doi.org/10.1145/3491101.3519717
VRySmart: a Framework for Embedding Smart Devices in Virtual Reality
Authors: Akhmajon Makhsadov, Donald Degraen, André Zenner, Felix Kosmalla, Kamila Mushkina, Antonio Krüger
Abstract: As immersive virtual experiences find their way into our living room entertainment, they are becoming part of our daily technological consumption. However, state-of-the-art virtual reality (VR) remains disconnected from other digital devices in our environment, such as smartphones or tablets. As context switches between acting in the virtual environment and resolving external notifications negatively influence immersion, we look towards integrating smart devices into virtual experiences. To this aim, we present the VRySmart framework. Through either optical marker tracking or simultaneous localization and mapping (SLAM), embedded smart devices can be used as VR controllers with different levels of integration while their content is incorporated into the virtual context to support the plausibility of the illusion. To investigate user impressions, we conducted a study (N = 10) where participants used a smartphone in four different virtual scenarios. Participants positively assessed smart device usage in VR. We conclude by framing implications for future work.
Citation
If you want to reference our work, please use the following bibtex entry:
@inproceedings{makhsadov2022vrysmart,
author = {Makhsadov, Akhmajon and Degraen, Donald and Zenner, Andr\'{e} and Kosmalla, Felix and Mushkina, Kamila and Kr\"{u}ger, Antonio},
title = {VRySmart: A Framework for Embedding Smart Devices in Virtual Reality},
year = {2022},
isbn = {9781450391566},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3491101.3519717},
doi = {10.1145/3491101.3519717},
abstract = { As immersive virtual experiences find their way into our living room entertainment, they are becoming part of our daily technological consumption. However, state-of-the-art virtual reality (VR) remains disconnected from other digital devices in our environment, such as smartphones or tablets. As context switches between acting in the virtual environment and resolving external notifications negatively influence immersion, we look towards integrating smart devices into virtual experiences. To this aim, we present the VRySmart framework. Through either optical marker tracking or simultaneous localization and mapping (SLAM), embedded smart devices can be used as VR controllers with different levels of integration while their content is incorporated into the virtual context to support the plausibility of the illusion. To investigate user impressions, we conducted a study (N = 10) where participants used a smartphone in four different virtual scenarios. Participants positively assessed smart device usage in VR. We conclude by framing implications for future work.},
booktitle = {CHI Conference on Human Factors in Computing Systems Extended Abstracts},
articleno = {358},
numpages = {8},
keywords = {smart devices, framework., virtual reality, haptic feedback},
location = {New Orleans, LA, USA},
series = {CHI EA '22}
}
License
(c) Copyright 2022 of the respective paper authors. For inquiries, please contact Donald Degraen (donald.degraen-at-gmail.com).
This software is made available under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
Attributions
https://sketchfab.com/3d-models/lightsaber-starwarshype-0dff87970ba44138982fcb132d3e667d