/navigation-2

A pratical work of navigation on Unity made for my class of Virtual Reality and Interaction

Primary LanguageC#

Navigation 2 - Practical Work

Navigation work at classes of Vrtual Reality and Interaction, (first part here)

The subject of the work is below :

1 Introduction

Question 1: You will work with two scenes in the hierarchy. A 3D scene already built (for example by importing: village.unitypackage) and your own scene in which you will include everything necessary for navigation and interaction. Only the latter will have to be rendered (export of the scene as a unity package).

Question 2. The goal is to obtain an effect similar to the following video (whatever the scene) :

https://www.fil.univ-lille1.fr/~aubert/rvi/movie/cadre.webm

It will therefore be in this tp:

  1. to grab a frame with the mouse.
  2. to retrieve the closest point to this frame.
  3. position the camera according to this nearest point.

2 Frame capture

Question 3. Make sure you have "First-Person" navigation on your main camera. Make sure that you can "disengage" this navigation to get the mouse cursor (which we will call "capture mode" in the following): when you left-click, you have the mouse cursor without navigation and another left-click returns to navigation mode.

Question 4: In input mode, code the input of a frame with the mouse by right click+move.

For the frame itself, create a game object (several possibilities: a cube may be suitable, but it doesn't matter).

The difficulty being to position this frame in the scene so that it corresponds to mouse (top-left corner of the frame at the time of down and bottom-right corner at the time of up must correspond to the mouse). It will be positioned in the plane of the screen, a little in front of the near to be visible.

For its positioning Plane.Raycast and Camera.ScreenPointToRay can help.

Put a transparent color to this frame to see through it.

Make sure that the capture is "perfect" whatever the orientation of the camera.

3 Render To Texture

The nearest point of the scene within this frame must be determined. This is based on the depth of the pixels of the scene within this frame.

The usual technique is to make a Render To Texture, then to recover the depth of the image produced. Unfortunately, if Unity does make a Render To Texture, it does not plan to read directly the depth produced (only the color produced).

Question 5: Make a second camera (we will call it "CameraCapture") to receive the Render To Texture. Make a RenderTexture rt; attribute (to draw in the frame buffer rt) and Texture2D capture; (to read the pixels of the frame buffer).

To create rt (once and for all in the start), just do rt = new RenderTexture (wrt, hrt, 16); where wrt and hrt are the width/height of your main camera (it is important that the view of this camera is identical to the view of the main camera : you can retrieve this with the pixelWidth and pixelHeight attributes of a camera).

We force this camera to draw in rt by simply doing cam.targetTexture = rt; (in start).

When updating, we can retrieve what is seen (i.e. what is in rt), by assigning capture with the pixels of rt :

    RenderTexture save = RenderTexture.active; // pour ne pas polluer le moteur Unity

    RenderTexture.active = rt;
    capture.ReadPixels (new Rect (0, 0, wrt, hrt), 0, 0); // copier les pixels du render to texture dans la texture capture
    capture.Apply (); // synchronisation

    RenderTexture.active = save;

To test, integrate into this camera:

  void OnGUI() {
    GUI.DrawTexture (new Rect (0, 0, 100,100), capture);
  }

You need to see what is seen by this camera (in its fixed position) in the top-left corner.

Question 6: This camera must have the same position/orientation as the main camera: do this in the update and you should see in the top-left corner exactly the same view as the main camera (in thumbnail).

Question 7. To use the values of the captured texture from the script you just have to retrieve all the values of the texture. For example, let's put in green color all the pixels of the captured frame :

  public void ChangeValue(Vector3 p1,Vector3 p2) {
  // p1, p2 : diagonale du cadre
  ... // déterminer x0,y0,w,h du cadre

    Color[] c = capture.GetPixels (x0, y0, w, h);
    for (int x = 0; x < w; ++x) {
      for (int y = 0; y < h; ++y) {
        c [y * w + x].g = 1.0f; // composante verte à 1
      }
    }

    capture.SetPixels (x0, y0, w, h,c); // affectation avec les couleurs modifiées
    capture.Apply (); // synchronisation
}

Code and test to see the colors of the thumbnail changed according to the entered frame (you have to call changeValue once the frame is entered).

4 Recovering the depth

If there is no problem to manipulate the color, you must be able to read the depth generated by the render to texture. Unity doesn't provide this operation, so you have to copy the depths as colors in a dedicated shader.

To be able to retrieve the depth generated by the main camera: cam.depthTextureMode = DepthTextureMode.Depth;. You can then use a dedicated texture in a shader. Make a shader with copy and paste :

Shader "Custom/shader2"
{
  subshader {
        Tags { "RenderType"="Opaque" }
    pass {

    CGPROGRAM
    #pragma vertex vert
    #pragma fragment frag
    #include "UnityCG.cginc"

      sampler2D _CameraDepthTexture; // la texture contenant les valeurs de profondeurs de la caméra principale (affectation automatique par Unity)

        struct outV {
          float4 pos : SV_POSITION;
          float4 fTex : TEXCOORD0;
        };

        struct inV {
          float4 position : POSITION;
        };



        outV vert(inV v) {
          outV res;

          res.pos = UnityObjectToClipPos(v.position);
          res.fTex=res.pos;
          return res;
        }

        float4 frag(outV v) : COLOR {
          float2 gett; // calculer screen coordinate dans [0,1]
          gett.x=(v.fTex.x/v.fTex.w)*0.5+0.5;
          gett.y=0.5-(v.fTex.y/v.fTex.w)*0.5;
          float f=tex2D(_CameraDepthTexture,gett.xy); // lire la valeur de depth
          return float4(f,0,0,1); // valeur de depth en couleur rouge
      }
    ENDCG
    }
  }
}

To force CameraCapture to use this shader for all objects in the scene: cam.SetReplacementShader (replace, "");. where replace is the shader created above.

Test: the thumbnail should appear in a red gradient corresponding to the depth values.

5 Nearest point

Question 8. Once you have entered the frame, simply retrieve the largest value (for the depth: 0 corresponds to the farthest, 1 to the nearest). Code this search for the largest by using ChangeValue as an inspiration. Make sure that this function returns the coordinates of the corresponding pixel + the depth.

Question 9. The depth value does not correspond to the distance to the camera (the stored depth corresponds to the interpolation of 1/z). To retrieve the distance to the camera, apply the following calculation in the script :

    zndc = 1.0f-readDepth; // valeur de depth trouvée
    float depthZ = far*near/(far-(far-near)*zndc); // distance à la caméra

Test the consistency of the found value (e.g. by displaying a sphere in the 3D scene corresponding to the found position).

6 Positioning the main camera

Question 10: Reposition the frame in 3D so that it corresponds to the value found in depth (the frame always has identical dimensions seen from the camera, but its depth corresponds to the point of the scene closest to the frame: see video).

Question 11. Position the camera so that its view corresponds to the frame (without first having the view corresponding to the dimension of the frame; then in a second step so that the width of the frame corresponds to the width of the view).