This repository contains code for a remote interactive surgery platform running on Micosoft HoloLens 2. For a detailed explanation of this software and evaluaation of our method, please refer to our paper Remote Interactive Surgery Platform (RISP): Proof of Concept for an Augmented-Reality-Based Platform for Surgical Telementoring.
Note: This prepository is still work in progress.
To use the voice communication functionality, it requires WebRTC signaler server. Please refer to this document's Install and run node-dss section
to setup the WebRTC signaler server, and make sure the server can take inbound connections from the Internet.
This software requires a Microsoft HoloLens 2 and a PC and those two devices can be reached to each other through a network.
- Unity 2019.4.28f1 with
Universal Windows Platform Build Support
andWindows Build Support (IL2CPP)
. - Visual Studio 2019 with
.NET desktop development
,Desktop development with C++
,Universal Windows Platform (UWP) development
, andGame development with Unity
. - Anaconda with
python 3.7
- Open
RISP-HL2
with Unity. - Open
RISP-HL
scene and insert the WebRTC signaler server address toManagers > WebRTCManager > NodeDssSignaler > Http Server Address
in the hierachy. - Build the scene with targeting
Universal Windows Platform
. - Open
package.appxmanifest
in the build folder and add the restricted capability to the manifest file.
<Package
xmlns:mp="http://schemas.microsoft.com/appx/2014/phone/manifest"
xmlns:uap="http://schemas.microsoft.com/appx/manifest/uap/windows10"
xmlns:uap2="http://schemas.microsoft.com/appx/manifest/uap/windows10/2"
xmlns:uap3="http://schemas.microsoft.com/appx/manifest/uap/windows10/3"
xmlns:uap4="http://schemas.microsoft.com/appx/manifest/uap/windows10/4"
xmlns:iot="http://schemas.microsoft.com/appx/manifest/iot/windows10"
xmlns:mobile="http://schemas.microsoft.com/appx/manifest/mobile/windows10"
xmlns:rescap="http://schemas.microsoft.com/appx/manifest/foundation/windows10/restrictedcapabilities"
IgnorableNamespaces="uap uap2 uap3 uap4 mp mobile iot rescap"
xmlns="http://schemas.microsoft.com/appx/manifest/foundation/windows10">
<Capabilities>
<rescap:Capability Name="perceptionSensorsExperimental" />
...
</Capabilities>
- Open
RISP-HL2.sln
in the build folder. - Deploy with Release / ARM (Not ARM64)
- Create a virtual environment of
python 3.7
usingAnaconda
. - Open
RISP-Com\main_application
and intall depenendices throughpip install -r requirements.txt
.
- Open
RISP-Com\voice_communicator
with Unity. - Open
Voice_client
scene and insert the WebRTC signaler server address toNodeDssSignaler > Http Server Address
. - Build the scene with targeting
PC, Mav & Linux Standalone
.
- Get your HoloLens's IP address.
- Launch
RISP-HL2
in your HoloLens. - Launch
main_application
bypython MainClient.py
inRISP-Com\main_application
. - Enter HoloLens's IP address.
- Launch built
Voice_Client
and clickCreateOffer
button.
- First version of remote interactive surgery platform code released.
- Add measurement tools (e.g., rulers in
RISP-HL2
). - Detailed intrcution of
How to use
. - etc.....
Please cite our paper.
@Article{jimaging9030056,
AUTHOR = {Kalbas, Yannik and Jung, Hoijoon and Ricklin, John and Jin, Ge and Li, Mingjian and Rauer, Thomas and Dehghani, Shervin and Navab, Nassir and Kim, Jinman and Pape, Hans-Christoph and Heining, Sandro-Michael},
TITLE = {Remote Interactive Surgery Platform (RISP): Proof of Concept for an Augmented-Reality-Based Platform for Surgical Telementoring},
JOURNAL = {Journal of Imaging},
VOLUME = {9},
YEAR = {2023},
NUMBER = {3},
ARTICLE-NUMBER = {56},
URL = {https://www.mdpi.com/2313-433X/9/3/56},
ISSN = {2313-433X},
DOI = {10.3390/jimaging9030056}
}
This code has been built with referencing, modifying, and using the following libraries / repositoies:
- Mixed Reality Toolkit
- HoloLensCameraStream
- HoloLens2-ResearchMode-Unity
- MixedReality-WebRTC
- Open3D
- OpenCV
- HoloLens2-Unity-ResearchModeStreamer
We thank authors of those codes and opensource community for making possible to build this code.