/3dtoolkit

Primary LanguageC++MIT LicenseMIT

3D Toolkit

A toolkit for building powerful stereoscopic 3d experiences that run on azure and stream to devices. 💪 👁️ ☁️

Note: If you are using Visual Studio 2017, ensure you have installed the v140 c++ build tools, and please do not update our projects when prompted to do so.

Grabbing latest prebuilt binaries

CI Builds - Not recommended

Pull Request CI Builds - Not recommended

Release/Stable Tagged Builds

Prerequisites

  • Windows 10 Anniversary Update / Windows Server 2012 R2 / Windows Server 2016
  • Visual Studio 2015 Update 3
  • Windows 10 SDK - 10.0.14393.795
  • Windows 10 DDK - If building WebRTC Library from source

Installing Prebuilt Libraries for /3dtoolkit-directx

After cloning this repository, run .\setup.cmd before opening the Project/Solution structure.

This will install and configure the following:

  • 32bit and 64bit Debug, Release, Exes, Dlls and PDBs from this commit Chromium m58 release
  • This patch will be applied to the above
  • 32bit and 64bit Debug and Release libraries for DirectX Toolkit
  • WebRTC-UWP M54 synced release for UWP-based clients (Hololens)

Note: We can't currently use the directxtk nuget packages because they don't provide static linking targets for release builds.

Once you see Libraries retrieved and up to date you may proceed and open the solution.

Build output

Build the solution from Visual Studio, and look in:

  • \3dtoolkit-directx\Samples\**\Build\** for server binaries.
  • \WebRTCNativeClient\Build\** for the native client binaries.
  • \WebRTCLibs\**\Exe\** for the sample signaling, turn and stun server binaries.

Building HoloLens Unity client

Note: Make sure you have Unity version 5.6.1f1 with UWP tools support.

You can find the HoloLens Unity client sample in \Samples\Client\Unity. The HoloLens client is using a native dll library for video decoding and rendering that needs to be built before you open the sample in Unity. Follow these steps:

  • Open 3DStreamingToolKit.sln from the root folder
  • Switch the build configuration to Release mode and x86.
  • Under Plugins -> UnityClient, find MediaEngineUWP and WebRtcWrapper projects.
  • Build MediaEngineUWP and WebRtcWrapper. This will copy the necessary dll's to the Unity project folder.
  • Open Unity and open the HoloLens client folder \Samples\Client\Unity
  • Under scenes folder, open 'StereoSideBySideTexture' scene.
  • Under StreamingAssets, you will find 'webrtcConfig.json'. Modify the values to match your signaling server and TURN server (optional, needed for VPN/Proxy networks)
  • In the scene hierarchy, open the Control gameobject and look at the Control Script.
    • Texture Width and Height: These settings are used to display the stereoscopic texture from the server. The client is expecting a left and right eye side-by-side texture and it will crop it for each eye. Since our servers are sending 720p quality, the default values are 2560 x 720 (1280 x 2 width for each eye). Change this if you are sending different resolutions.
    • Frame Rate: This is the expected frame rate from the server. The native engine will be optimized to maintain this framerate. 30 is the currently recommended value, going above this, can cause loss of quality and corrupt frames.

Note: Make sure you set the same frame rate on the server side!

  • Go to File -> Build Settings and select Windows Store platform. If that is not available, you need to add UWP support to your Unity installation.
  • Select
    • SDK: Universal 10
    • Target Device: HoloLens
    • UWP Build type: D3D
    • UWP SDK: 10.0.14393
  • Build the project to an empty folder and open the generated UnityX86.sln.
  • Switch configuration to Release mode and x86.
  • Deploy to your machine or HoloLens. Make sure the server is running, the app will automatically connect to the ip address that was set in the step above.
  • Open the desired server and connect to the Unity client.

JSON Configuration Options for Native Client and Native Servers:

Use webrtcConfig.json for TURN relay communication

Use webrtcConfigStun.json for STUN P2P communication

At a minimum, you will need to deploy an instance of the signaling server below and use the STUN configuration file to test the native client and server. You can run client, server and signaling server from the same development machine.

nvEncConfig.json is the nvencode configuration file.

  • Set "useSoftwareEncoding" to true to use the CPU for video encoding - this will increase latency and should only be used for development on computers without an nvidia video card.
  • Set "serverFrameCaptureFPS": 60 and "fps" to the same value. Maximum FPS is dependent on card and scene complexity.
  • Set "bitrate" and "minBitrate" to set the max and min bitrates for video streaming - recommended maximum @ 10mbps and min to 5.5mbps for high quality streaming. Anything over 10mbps will not visibly increase quality (see test runners for validation). Setting minBitrate to 0 enables webrtc to drop bitrate to whatever it needs to in order to keep video streaming fluidly.

Distributing binaries

Note: Currently for others you'll need to zip and upload yourself to match the .\WebRTCLibs\webrtcInstallLibs.ps1 script.

https://3dtoolkitstorage.blob.core.windows.net/libs/m58patch_headers.zip https://3dtoolkitstorage.blob.core.windows.net/libs/m58patch_Win32.zip https://3dtoolkitstorage.blob.core.windows.net/libs/m58patch_x64.zip

Standing up production TURN/STUN/Signaling Servers

Building WebRTC Libraries from Source (You don't need to do this unless you are changing the encoder)

Note: If you want to build webrtc yourself, you will need to install Visual Studio 2015 with Update 3.

Visual Studio 2015 Update 3 Community Edition ISO

Visual Studio 2015 Update 3 Professional Edition ISO

Visual Studio 2015 Update 3 Enterprise Edition ISO

Be sure to install all of the C++ language tools and the Windows Universal Components.

Once finished installing, install the Windows DDK version 1607 as it contains debugging tools needed by WebRTC.

Finally, launch PowerShell with Administrative persmissions and run

.\WebRTCLibs\webrtcSetup.ps1 -WebRTCFolder C:\path\here

(If path is omitted, it will install to C:\WebRTCSource. Be careful with long path names, depot tools may fail to install.)

This will automate all of the following steps (from here):

  1. Set environment variables
  2. Download and install pre-requisites
  3. Create webrtc repo from m58 Chromium release
  4. Cleans repo
  5. Create build definitions for Win32/64 Debug and Release
  6. Run the builds for each configuration
  7. Creates Dist folder structure
  8. Moves all packages data to dist
  9. (TODO) Auto-zip and upload to blob store

Once finished building, for local use, copy the contents of

C:\<path to source>\webrtv-checkout\dist

to the WebRTCLibs folder in this repo.

!IMPORTANT - USING UNITY AS A 3D VIDEO STREAMING SERVER IS AGAINST THE SOFTWARE TERMS OF SERVICE - THE UNITY SERVER SAMPLE IS PROVIDED FOR DEMO AND EDUCATIONAL PURPOSES ONLY. CONTACT UNITY FOR LICENSING IF YOU WANT TO USE THE UNITY SERVER SAMPLE IN ANY COMMERCIAL ENVIRONMENT

Please refer to https://unity3d.com/legal/terms-of-service/software

Streaming and Cloud Gaming Restrictions

You may not directly or indirectly distribute Your Project Content by means of streaming or broadcasting where Your Project Content is primarily executed on a server and transmitted as a video stream or via low level graphics render commands over the open Internet to end user devices without a separate license from Unity. This restriction does not prevent end users from remotely accessing Your Project Content from an end user device that is running on another end user device.