The purpose of this repository is to provide a reproducible (possibly isolated) environment for high level robotics software. Currently the main concern of this repo is interacting with Intel RealSense Cameras. This environment (code of this repo) can be be distributed as a docker image that is cross-compiled to both x86-64 and arm64 architectures (Regular computers as well as raspberry pi). Here is a quick list of thing this repository is meant to do:
- Have a stable pre-setup ubuntu environment in a docker image with all installed dependencies.
- Communicate with Intel RealSense Devices using librealsense SDK 2.0.
- Provide a communication framework using ECAL which provides efficient shared memory to publish "messages".
- Provide simple configurations file that can configure publishing of realsense cameras and saving data.
- See
rspub_default.tomlfor publishing andrssave_default.tomlfor saving RealSense data.
- See
- Provide Python client code for visualizing mesh creation.
All dependencies and installation procedures can be found in Docker/base/Dockerfile. You can use docker and have everything taken care or you can follow the installation procedures in the file. Here is a basic summary of the environment:
- Ubuntu Base Image 18.04
- Python 3 w/ Scipy, numpy
- CMake 3.15.5 - Needed because realsense asks for 3.11 or higher.
- Open CV 3.4.7 with user contributed modules
- ECAL for marshalling and communication
- Protobuf - Serialization format (Thinking of changing to Flatbuffers eventually)
- RealSense SDK
- Open3D - new point cloud and mesh processing library from Intel
- GFLAGS and GLOG for command line parsing and logging
If you want to install locally please execute the following scripts (there will need to be some adjustments for you environment)
./Docker/base/dependencies_local.sh./Docker/base/ecal_local.sh./Docker/base/opencv_local.sh./Docker/base/realsense_local.sh- Uses RSUSB backend, not kernel
This is only required if you want two computers on the same network to talk to each other.
- Update each computers hosts file so that they are aware of each other. An example below. Don't forget the
.localdomain.
Example:
127.0.0.1 localhost
::1 localhost
127.0.1.1 cerberus.localdomain cerberus
192.168.1.3 raspberrypi
- Configure Multicast for device -
ifconfig YOUR_NETWORK_DEVICE multicast - Add Multicast route -
sudo route add -net 224.0.0.0 netmask 255.0.0.0 dev YOUR_NETWORK_DEVICE
Note that I am using the non-default multi-cast group 224.0.0.0 for ecal because my router was for some reason blocking the default.
When having any problems first see these issues: Issue 1, Issue 2
The software is configured to use the config/ecal/ecal.ini file. Note that some values have been changed from the default. If you are using ecal_mon_gui on an external computer to monitor messages dont forget to modify your ecal.ini to point to the correct UDP multicast address and other options (e.g., ttl).
- Install Docker
git clone --recursive https://github.com/JeremyBYU/realsense-tracking.git && cd realsense-tracking
A raspberry pi image has already been created and set up. Download the image and flash the sd card.
uname: a2sys
password: a2sysblah
ssh -X a2sys@192.168.1.25- Allows XForwarding if needed. Might need to use gui (hdmi) to find out what the ip is first. IP subject to change for your own network.cd $HOME/Documents/realsense-tracking
rs-pose,rs-enumerate-devices- Need to "open" the sensors on host first before launching docker. Not sure why.docker run --rm --privileged -it --env="DISPLAY" --net=host --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --mount type=bind,source="$(pwd)",target=/opt/workspace --name realsense jeremybyu/realsense:buildx- Raspberry PI- Optional -
rm -rf build && mkdir build && cd build && cmake .. && make && cd ..
Alternative X86 Launch - docker run --rm --privileged -it --env="DISPLAY" --net=host --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --mount type=bind,source="$(pwd)",target=/opt/workspace --name realsense --user $(id -u):$(id -g) --ipc=host jeremybyu/landing:latest
Also see Docker/README.md for building and launching.
Install all dependencies and simply cd to realsense-tracking
Make a cmake build directory. I use:
cmake-buildfor local development. No docker.dk-x86_64-buildfor x86 Docker development. On your local computer, inside docker x86 container, building application.dk-aarch64-buildfor arm Docker development. On your local computer, inside docker arm container (using QEMU underneath), building application.
- If using docker, enter into terminal shell
mkdir MY_BUILD_DIR && cd MY_BUILD_DIRcmake .. && cmake --build -j8
All log files should be saved in the logs directory. Simply add --log_dir=./logs to command line of any application below.
You can view the log files with: less +F logs/rs-pub.INFO
Your can clear log files with: ./scripts/misc/clear_logs.sh
Will publish topics which are configured in a toml file. Configured by config/rspub_default.toml.
GLOG_logtostderr=1 ./bin/rs-pub
./bin/x86_64/rs-pub --log_dir=./logs --v=1 --config=config/l515/rspub_default.toml
--force_udp is an option
Will Subscribe to topics and Save data to disk. Configured by config/rssave_default.toml.
GLOG_logtostderr=1 ./bin/rs-save --config=config/rssave_default.toml
Will convert saved protofiles in a folder to text format. This is for any point clouds or pose information.
python scripts/rs-proto-folder.py
Creates an RPC Server that allows users to generate meshes on demand with simple requests. This requires both the T265 Poses and D4XX RGBD frames to be published.
It will automatically subscribe to RGBD Images and integrate them to a voxel volume. Users can (on demand) request
to extract meshes from the voxel volume. Configured by config/rsintegrate_default.toml.
export OMP_NUM_THREADS=1- Multithreading actually screws this worse!GLOG_logtostderr=1 ./bin/rs-integrate-server --v=1
- https://stackoverflow.com/a/19733677
- Enter this into terminal 1
socat -d -d pty,raw,echo=0 pty,raw,echo=0 - Now
/dev/pts/1and/dev/pts/2are linked. Tested with writing to/dev/pts/2and reading from/dev/pts/1
Instead of deploying a docker image, you can just deploy the binaries it built! The following script will gather all the dependencies for the binary in the bin/$ARCH folder.
You must be inside the docker container when executing this script
./scripts/dependencies/gather_all.sh
You can launch the binaries in parallel with this command: parallel --verbose --linebuffer ::: scripts/launch/launch_rspub.sh scripts/launch/launch_integrate.sh
export LD_LIBRARY_PATH=./bin/x86_64GLOG_logtostderr=1 ./bin/x86_64/ld-linux-x86-64.so.2 ./bin/rs-pub --help.
A couple of caveats when running binaries in your envrionment: If you are using raspbian and have the 64bit kernel mode enabled then you must use the loader in the dependencies folder to launch the program. Basically
export LD_LIBRARY_PATH=./bin/aarch64GLOG_logtostderr=1 ./bin/aarch64/ld-linux-aarch64.so.1 ./bin/rs-pub --help.
Just tested this on ARM and it actually worked! The only thing that was missing was libecaltime-localtime.so.
You can save a docker image to a file as so:
docker save -o
docker save -o docker_images/roboenv.tar jeremybyu/realsense:latest
You can load a docker image as so: docker load -i roboenv.tar
scp pi@192.168.1.3:/home/pi/Documents/realsense-tracking/data/Default.ply data/Default.ply
The image is 3.1 GB. Open3D is about 1.1 GB! 475 MB of that is the python installs from extension (338 MB is just the open3d.so file). Maybe get rid of it?
Cmp Size Command Permission UID:GID Size Filetree
63 MB FROM d9bd0dcaa40bc11 drwx------ 0:0 41 MB ├── root
988 kB [ -z "$(apt-get indextargets)" ] drwx------ 0:0 41 MB │ └── .cache
745 B set -xe && echo '#!/bin/sh' > /usr/sbin/policy-rc.d && echo 'exit 101' >> /usr/sbin/p drwx------ 0:0 41 MB │ └─⊕ pip
7 B mkdir -p /run/systemd && echo 'docker' > /run/systemd/container drwxr-xr-x 0:0 1.1 GB └── usr
3.2 MB echo 'Etc/UTC' > /etc/timezone && ln -s /usr/share/zoneinfo/Etc/UTC /etc/localtime & drwxr-xr-x 0:0 1.1 GB └── local
446 MB apt-get update && apt-get install -q -y bash-completion lsb-release python3- drwxr-xr-x 0:0 5.3 kB ├─⊕ bin
0 B #(nop) WORKDIR /opt/workspace drwxr-xr-x 0:0 140 B ├─⊕ etc
322 MB apt-get update && apt-get install -q -y wget git libssl-dev libusb-1.0-0-dev pkg-con drwxr-xr-x 0:0 13 MB ├─⊕ include
156 MB apt-get install --no-install-recommends -y build-essential libgtk2.0-dev pkg-config liba drwxr-xr-x 0:0 1.1 GB ├── lib
48 MB apt-get install --no-install-recommends -y graphviz build-essential zlib1g-dev libhdf5-d drwxr-xr-x 0:0 3.0 kB │ ├─⊕ cmake
67 MB wget -O /opt/cmake-3.15.5.tar.gz https://cmake.org/files/v3.15/cmake-3.15.5.tar.gz && -rw-r--r-- 0:0 597 MB │ ├── libOpen3D.a
745 B #(nop) COPY file:03028ab537c33176c2c9827484d3abff128258e6c117cf751ff343c7b0df99dc in /tm -rw-r--r-- 0:0 4.7 MB │ ├── libjsoncpp.a
745 B chmod u+x /tmp/opencv.sh -rw-r--r-- 0:0 5.6 MB │ ├── libqhullcpp.a
192 MB git clone -b '3.4.7' --single-branch https://github.com/opencv/opencv.git /opt/opencv && -rw-r--r-- 0:0 2.3 MB │ ├── libqhullstatic_r.a
40 MB cd /opt && git clone --recursive git://github.com/continental/ecal.git && cd eca -rw-r--r-- 0:0 854 kB │ ├── libtinyfiledialogs.a
395 MB git clone https://github.com/IntelRealSense/librealsense.git /opt/librealsense && cd /op -rw-r--r-- 0:0 3.1 MB │ ├── libtinyobjloader.a
0 B ln -s /usr/bin/python3 /usr/bin/python & ln -s /usr/bin/pip3 /usr/bin/pip -rw-r--r-- 0:0 765 kB │ ├── libturbojpeg.a
1.2 GB cd /opt && git clone --recursive https://github.com/intel-isl/Open3D && cd /opt/Open drwxrwxr-x 0:50 495 MB │ └── python3.6
3.4 MB apt-get install --no-install-recommends -y nano htop drwxrwxr-x 0:50 495 MB │ └─⊕ dist-packages
32 MB apt-get install --no-install-recommends -y libgflags-dev libgoogle-glog-dev gfortran drwxr-xr-x 0:0 4.6 MB └─⊕ share