Digital PTZ Support (Lossless Zoom)
Luxonis-Brandon opened this issue · 5 comments
Start with the why:
DepthAI and megaAI have a 12MP color sensor build in (and the FFC version can be used w/ the 12MP Pi HQ Camera). Often lower-resolution encoded video is wanted, but perhaps not necessarily of the full frame - and perhaps only for a particular region of interest.
For example, using the full 12MP input resolution of 4056x3040, the following 'lossless' zooms are possible:
- 4K (3840x2160): 1.48:1
- 1080p (1920x1080): 5.95:1
- 720p (1280x720): 13.37:1
So that's up to a 'lossless' zoom of ~13x when wanting 720p video encoding output. By lossless, I mean the capability to still have direct pixel-to-pixel mapping, so it is indeed still a full/native resolution output, direct from actual pixels.
Move to the how:
The internal frameworks we use for convert/crop/rescale off of the full resolution of the image sensor could be changed to allow an initial X,Y (in pixels) crop location and rework of our encoding implementation to support zoom factor (up to the max direct-pixel-map) when cropping/scaling by changing it to support runtime change of resolution.
Move to the what:
- Support Digital PTZ (pan, tilt, zoom) video output of 4K, 1080p, or 720p.
Note: It may also be possible to support zooming past the lossless pixel-to-pixel direct mapping zoom limits (above) using hardware-accelerated Lanczos or bilinear interpolation, if this is of interest. (To for example, allow zooming to 15x at 1080p output resolution and interpolating between the actual pixels to produce the full resolution encoded video).
As an update, the first version of this will be a fixed-zoom level, with the capability to pan/tilt around the whole frame (so a moving window).
And here is the example of how to use it in C++:
https://github.com/luxonis/depthai-core/blob/gen2_develop/examples/src/color_camera_control_example.cpp
And as a note on the example above:
It crops out a 640x360 region from RGB camera and resizes that to 300x300 (preview).
These parameters can be changed to fit your needs. For example to output 1920x1080 crop from a 4K image sensor setting, set colorCam.setVideoSize(1920, 1080) and set camera resolution to 4k:
colorCam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K)
And to set the sensor to 12 MP, use SensorResolution.THE_12_MP instead of SensorResolution.THE_4_K.
both links are not working! could you please share if possible