PerspectiveProjection
PerspectiveProjection is a Unity package intended for camera frustum projection, a Technique that is used in virtual production to turn LED screens into a backdrop where CG content is projected in real-time. It can be used to give depth to flat screens when captured by a real-world camera which position and rotation are tracked in some way (i.e. Vive Tracker) and passed to a virtual camera within a Unity scene.
The preview below shows a possible result seen from the point-of-view of the recording camera (top half) and from an arbitrary point-of-view to illustrate the overall setup (bottom half) - LED Screen as backdrop to physical "actors" and a tracked camera (in this instance, an HTC Vive Tracker attached to a phone).
Notes
- Tested with Unity 2021 LTS HDRP, in Editor and Play mode, on a gaming laptop with an RTX 3080 graphics card.
- Intended for projection onto one single screen (might work with multiple ones with some extra work).
- This solution does not rely on any particular camera tracking method which must be implemented separately.
- It is NOT intended for production work, is an explorative project and it hasn't been thoroughly tested.
Setup
Installation
- Disable existing cameras in the scene.
- Create a new layer called "PerspectiveProjection".
- Import PerspectiveProjection.unitypackage into a project.
- Locate and select the ScreenProjection and CameraOrigin prefabs (PerspectiveProjection > Prefabs) and place both of them in the scene.
Checklist
- The hierarchy of the gameobjects should look as follows.
- Layer should be set to PerspectiveProjection for all gameobjects with the exception of Debug. Debug should be deactivated unless you need to see the virtual representation of the physical screen (ScreenProjection) in the Game window for debug purposes.
- One material should be assigned to ScreenProjection (ScreenProjectionMat).
- One script should also be assigned to ScreenProjection linking properties to the CameraOrigin, TrackedCamera and Display Camera gameobjects as follows.
Scale and Resolution
- The ScreenProjection prefab contains a quad that represents the physical screen position and dimension relative to the virtual world. The visible side of the quad represents the front of the physical screen. Use the transform gizmo for positioning the ScreenProjection prefab where preferred.
Note: DO NOT try to resize it by rescaling it (the attached script will prevent so anyways), instead do the following to set resolution and scale:
- In the Project folder, locate and select the RenderTexture named PerspectiveProjectionRT (PerspectiveProjection > Textures) and set its size in the Inspector to match the aspect ratio of the physical screen (i.e. 1920x1080 for a Full HD 16:9 screen).
- Set the Game window resolution to the same size or aspect ratio (i.e. 1920x1080 or 16:9).
- In the Hierarchy, select ScreenProjection and in the Inspector set the Resolution property (Perspective Projection Script) with the same values used for the RenderTexture (i.e. 1920x1080).
Then, set screen real-world size and scene scale as follows:
- With ScreenProjection still selected, change the Screen Height property to match the physical screen real-world height in meters (i.e. 32cm = 0.32).
- If you need to change the scale of ScreenProjection relative to the virtual world, set the Scale Factor property value to a different one. This is useful if your scene content is not following 1 unit = 1 meter or if you simply want the scene to be projected smaller or bigger onto the physical screen.
Note: By rescaling PerspectiveProjection following the above steps, you will notice that CameraOrigin scale values will also change accordingly. This ensures that the tracked camera movements in the real-world are properly scaled in the Unity scene.
Camera Tracking and Positioning
This package should work with any tracking method (i.e. Vive Tracker) that allows to mirror the position of a real-world camera to that of the TrackedCamera gameobject in the scene.
- To position the camera in relation to the screen (ScreenProjection) move and rotate CameraOrigin as needed.
- Add the necessary component for tracking the real-world position of the camera to the Tracker gameobject (i.e. SteamVR Tracked Object). The position and rotation of Tracker should be zero since it will be overwritten by the added component anyways.
- It is unlikely that the position of the tracker device (i.e. Vive Tracker) and the camera's lens are exactly aligned. Apply necessary offset values (in meters) to the TrackedCamera gameobject.
Camera FOV
- Select TrackedCamera and change its Field of View property to match that of the physical camera.
- If you don't know the FOV of your camera you can approximate a value and fine tune it later when projecting on the screen.
Important Notes:
- DO NOT try to manually position the camera by moving TrackedCamera, this should only be use for offset values between the physical tracker and the camera lens. ALWAYS use the parent gameobject CameraOrigin for manual positioning.
- DO NOT move TrackedCamera outside of Tracker or the latter outside of CameraOrigin. This ensures that the tracked camera movements are scaled and offset correctly.
- If for any reason TrackedCamera must be replaced by another camera gameobject, this must be made a child of Tracker. Also its Target Texture property must be set to the PerspectiveProjectionRT RenderTexture (PerspectiveProjection > Textures).
Projecting on a physical screen
- Set the Game window at full screen on the target physical display. To remove the top bar there's a handy tool available on the Unity Asset Store called "Fullscreen Editor''. Note that this is a paid third party tool - use it at your own risk.
- Once the Tracked gameobject starts getting real-world position and rotation from the tracker device (i.e. Vive Tracker) you might need to offset its position in the editor so that they are "aligned". Do so by modifying the position and rotation of CameraOrigin.
- It's important to correctly match the relative distance between the camera and the screen between real-world and the Unity scene.
