Send RGB-D images over spout / syphon / NDI with visiongraph.
Source: Intel® RealSense™ Sample Data
It is recommended to use Python 3.11 (3.8, 3.9 or 3.10 should work too) and should run on any OS. First create a new virtualenv and activate it.
After that install all dependencies:
pip install space-streamTo be able to use a ZED Camera, please follow the tutorial on the ZED Python API website.
- Install the ZED SDK (together with CUDA)
- Run the command
python "C:\Program Files (x86)\ZED SDK\get_python_api.py"inside the virtual python environment
Simply run the spacestream module with the following command to run a capturing pipeline (RealSense based). After that you can open a spout receiver / syphon receiver and check the result there.
space-stream --input realsense
To use the Azure Kinect use the azure input type:
space-stream --input azure
By default, either Syphon (MacOS) or Spout (Windows) is used to send the image to other applications. Since 0.3.0 it is possible to send the image over NDI by using the --ndi argument:
space-stream --input azure --ndi
To control the settings over OSC, start the application with the --osc argument. Please, listen for changes on port 7400 and to send changes, use port 7401 (by default).
space-stream --input azure --osc
/space-stream/record (Bidirectional): bool
/space-stream/codec (Bidirectional): DepthCodecType
/space-stream/min_distance (Bidirectional): float
/space-stream/max_distance (Bidirectional): float
/space-stream/depth_rectification (Bidirectional): bool
/space-stream/cam_auto_exposure (Bidirectional): bool
/space-stream/cam_exposure (Bidirectional): int
/space-stream/cam_iso (Bidirectional): int
/space-stream/cam_auto_white_balance (Bidirectional): bool
/space-stream/cam_white_balance (Bidirectional): int
/space-stream/masking (Bidirectional): bool
/space-stream/stream_name (Bidirectional): str
python setup.py distributeTo develop with this project, clone the git repository and install the dependencies from the requirements:
pip install -r requirements.txtTo call the module directly, use the -m command from python:
python -m spacestream
By default the depthmap is encoded by the linear codec. It is possible to change the behaviour to use a specific encoding method. Be aware that some functions have an impact on performance. Here is a list of all available codecs:
Linear,
UniformHue
InverseHue
The codecs UniformHue and InverseHue are implemented according to the Intel whitepaper about Depth image compression by colorization.
The encoded bit-depth depends on the codec used. For Linear codec there are two different bit-depths encoded. First the 8-bit encoding in the red channel and 16-bit encoded values in the green (MSB) and blue (LSB) channel.
To define the min and max distance to encode, use the --min-distance and --max-distance parameter.
usage: space-stream [-h] [-c CONFIG] [-s SETTINGS]
[--loglevel {critical,error,warning,info,debug}]
[--record RECORD]
[--codec Linear, UniformHue, InverseHue, RSColorizer]
[--min-distance MIN_DISTANCE]
[--max-distance MAX_DISTANCE] [--stream-name STREAM_NAME]
[--input video-capture,image,realsense,azure,camgear,zed]
[--input-size width height] [--input-fps INPUT_FPS]
[--input-rotate 90,-90,180] [--input-flip h,v]
[--input-mask INPUT_MASK] [--input-crop x y width height]
[--raw-input] [--channel CHANNEL]
[--input-skip INPUT_SKIP]
[--input-backend any,vfw,v4l,v4l2,firewire,fireware,ieee1394,dc1394,cmu1394,qt,unicap,dshow,pvapi,openni,openni_asus,android,xiapi,avfoundation,giganetix,msmf,winrt,intelperc,openni2,openni2_asus,gphoto2,gstreamer,ffmpeg,images,aravis,opencv_mjpeg,intel_mfx,xine]
[-src SOURCE] [--input-path INPUT_PATH]
[--input-delay INPUT_DELAY] [--exposure EXPOSURE]
[--gain GAIN] [--white-balance WHITE_BALANCE] [--depth]
[--depth-as-input] [-ir] [--rs-serial RS_SERIAL]
[--rs-json RS_JSON] [--rs-play-bag RS_PLAY_BAG]
[--rs-record-bag RS_RECORD_BAG] [--rs-disable-emitter]
[--rs-bag-offline]
[--rs-auto-exposure-limit RS_AUTO_EXPOSURE_LIMIT]
[--rs-auto-gain-limit RS_AUTO_GAIN_LIMIT]
[--rs-filter decimation,spatial,temporal,hole-filling [decimation,spatial,temporal,hole-filling ...]]
[--rs-color-scheme Jet,Classic,WhiteToBlack,BlackToWhite,Bio,Cold,Warm,Quantized,Pattern]
[--k4a-align-to-color] [--k4a-align-to-depth]
[--k4a-device K4A_DEVICE] [--k4a-depth-clipping min max]
[--k4a-ir-clipping min max] [--k4a-play-mkv K4A_PLAY_MKV]
[--k4a-record-mkv K4A_RECORD_MKV]
[--k4a-depth-mode OFF,NFOV_2X2BINNED,NFOV_UNBINNED,WFOV_2X2BINNED,WFOV_UNBINNED,PASSIVE_IR]
[--k4a-passive-ir]
[--k4a-color-resolution OFF,RES_720P,RES_1080P,RES_1440P,RES_1536P,RES_2160P,RES_3072P]
[--k4a-color-format COLOR_MJPG,COLOR_NV12,COLOR_YUY2,COLOR_BGRA32,DEPTH16,IR16,CUSTOM8,CUSTOM16,CUSTOM]
[--k4a-wired-sync-mode STANDALONE,MASTER,SUBORDINATE]
[--k4a-subordinate-delay-off-master-usec K4A_SUBORDINATE_DELAY_OFF_MASTER_USEC]
[--midas] [--mask]
[--segnet mediapipe,mediapipe-light,mediapipe-heavy]
[--parallel] [--num-threads NUM_THREADS] [--no-fastmath]
[--no-filter] [--no-preview] [--record-crf RECORD_CRF]
[--view-pcd] [--view-3d] [--osc] [--osc-host OSC_HOST]
[--osc-in-port OSC_IN_PORT] [--osc-out-port OSC_OUT_PORT]
[--ndi]
RGB-D framebuffer sharing demo for visiongraph.
options:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
Configuration file path.
-s SETTINGS, --settings SETTINGS
Settings file path (json).
--loglevel {critical,error,warning,info,debug}
Provide logging level. Example --loglevel debug,
default=warning
--record RECORD Record output into recordings folder.
--codec Linear, UniformHue, InverseHue, RSColorizer
Codec how the depth map will be encoded.
--min-distance MIN_DISTANCE
Min distance to perceive by the camera.
--max-distance MAX_DISTANCE
Max distance to perceive by the camera.
--stream-name STREAM_NAME
Spout / Syphon stream name.
input provider:
--input video-capture,image,realsense,azure,camgear,zed
Image input provider, default: video-capture.
--input-size width height
Requested input media size.
--input-fps INPUT_FPS
Requested input media framerate.
--input-rotate 90,-90,180
Rotate input media.
--input-flip h,v Flip input media.
--input-mask INPUT_MASK
Path to the input mask.
--input-crop x y width height
Crop input image.
--raw-input Skip automatic input conversion to 3-channel image.
--channel CHANNEL Input device channel (camera id, video path, image
sequence).
--input-skip INPUT_SKIP
If set the input will be skipped to the value in
milliseconds.
--input-backend any,vfw,v4l,v4l2,firewire,fireware,ieee1394,dc1394,cmu1394,qt,unicap,dshow,pvapi,openni,openni_asus,android,xiapi,avfoundation,giganetix,msmf,winrt,intelperc,openni2,openni2_asus,gphoto2,gstreamer,ffmpeg,images,aravis,opencv_mjpeg,intel_mfx,xine
VideoCapture API backends identifier., default: any.
-src SOURCE, --source SOURCE
Generic input source for all inputs.
--input-path INPUT_PATH
Path to the input image.
--input-delay INPUT_DELAY
Input delay time (s).
--exposure EXPOSURE Exposure value (usec) for depth camera input (disables
auto-exposure).
--gain GAIN Gain value for depth input (disables auto-exposure).
--white-balance WHITE_BALANCE
White-Balance value for depth input (disables auto-
white-balance).
--depth Enable RealSense depth stream.
--depth-as-input Use colored depth stream as input stream.
-ir, --infrared Use infrared as input stream.
--rs-serial RS_SERIAL
RealSense serial number to choose specific device.
--rs-json RS_JSON RealSense json configuration to apply.
--rs-play-bag RS_PLAY_BAG
Path to a pre-recorded bag file for playback.
--rs-record-bag RS_RECORD_BAG
Path to a bag file to store the current recording.
--rs-disable-emitter Disable RealSense IR emitter.
--rs-bag-offline Disable realtime bag playback.
--rs-auto-exposure-limit RS_AUTO_EXPOSURE_LIMIT
Auto exposure limit (ms).
--rs-auto-gain-limit RS_AUTO_GAIN_LIMIT
Auto gain limit (16-248).
--rs-filter decimation,spatial,temporal,hole-filling [decimation,spatial,temporal,hole-filling ...]
RealSense depth filter.
--rs-color-scheme Jet,Classic,WhiteToBlack,BlackToWhite,Bio,Cold,Warm,Quantized,Pattern
Color scheme for depth map, default: WhiteToBlack.
--k4a-align-to-color Align azure frames to color frame.
--k4a-align-to-depth Align azure frames to depth frame.
--k4a-device K4A_DEVICE
Azure device id.
--k4a-depth-clipping min max
Depth input clipping.
--k4a-ir-clipping min max
Infrared input clipping.
--k4a-play-mkv K4A_PLAY_MKV
Path to a pre-recorded bag file for playback.
--k4a-record-mkv K4A_RECORD_MKV
Path to a mkv file to store the current recording.
--k4a-depth-mode OFF,NFOV_2X2BINNED,NFOV_UNBINNED,WFOV_2X2BINNED,WFOV_UNBINNED,PASSIVE_IR
Azure depth mode, default: NFOV_UNBINNED.
--k4a-passive-ir Use passive IR input.
--k4a-color-resolution OFF,RES_720P,RES_1080P,RES_1440P,RES_1536P,RES_2160P,RES_3072P
Azure color resolution (overwrites input-size),
default: RES_720P.
--k4a-color-format COLOR_MJPG,COLOR_NV12,COLOR_YUY2,COLOR_BGRA32,DEPTH16,IR16,CUSTOM8,CUSTOM16,CUSTOM
Azure color image format, default: COLOR_BGRA32.
--k4a-wired-sync-mode STANDALONE,MASTER,SUBORDINATE
Synchronization mode when connecting two or more
devices together, default: STANDALONE.
--k4a-subordinate-delay-off-master-usec K4A_SUBORDINATE_DELAY_OFF_MASTER_USEC
The external synchronization timing.
--midas Use midas for depth capture.
masking:
--mask Apply mask by segmentation algorithm.
--segnet mediapipe,mediapipe-light,mediapipe-heavy
Segmentation Network, default: mediapipe.
performance:
--parallel Enable parallel for codec operations.
--num-threads NUM_THREADS
Number of threads for parallelization.
--no-fastmath Disable fastmath for codec operations.
debug:
--no-filter Disable realsense image filter.
--no-preview Disable preview to speed.
--record-crf RECORD_CRF
Recording compression rate.
--view-pcd Display PCB preview (deprecated, use --view-3d).
--view-3d Display PCB preview.
osc:
--osc Enable OSC support for settings.
--osc-host OSC_HOST OSC host address (default: 0.0.0.0)
--osc-in-port OSC_IN_PORT
OSC receiving port address (default: 7401)
--osc-out-port OSC_OUT_PORT
OSC receiving port address (default: 7400)
output:
--ndi Use NDI for frame buffer sharing.
Args that start with '--' can also be set in a config file (specified via -c).
Config file syntax allows: key=value, flag=true, stuff=[a,b,c] (for details,
see syntax at https://goo.gl/R74nmi). In general, command-line values override
config file values which override defaults.
Copyright (c) 2024 Florian Bruggisser