twitchslam

A toy implementation of monocular SLAM written while livestreaming.

Stream

https://www.twitch.tv/tomcr00s3

Will be streaming again tomorrow. May 28, 2018

By the end of the day, twitchslam will be decent.

Usage

export REVERSE=1   # Hack for reverse video
export F=500       # Focal length (in px)

./slam.py <video.mp4>

# good example
F=525 ./slam.py videos/test_freiburgxyz525.mp4

# ground truth
F=525 ./slam.py videos/test_freiburgrpy525.mp4 videos/test_freiburgrpy525.npz

# kitti example
REVERSE=1 F=984 ./slam.py videos/test_kitti984_reverse.mp4

# extract ground truth
tools/parse_ground_truth.py videos/groundtruth/freiburgrpy.txt videos/test_freiburgrpy525.npz

Classes

Frame -- An image with extracted features
Point -- A 3-D point in the Map and it's 2-D Frame correspondences
Map -- A collection of points and frames
Display2D -- SDL2 display of the current image
Display3D -- Pangolin display of the current map

Libraries Used

SDL2 for 2-D display
cv2 for feature extraction
pangolin for 3-D display
g2opy for optimization (soon!)

Rendered Scene Test

./fakeslam.py

NOTE: The test currently doesn't work reliably. It seems adding a small amount of Gaussian noise to the point positions can cause the optimizer to fall into really bad local minima. This may just be caused by poor initialization, as I'm not sure how stable Essential matricies are.

TODO: Investigate the stability of Essential matrix recovery.

DONE

BUGFIX: Why is window initting small? (fixed? ish, zoom is broken sometimes)
BUGFIX: Turning doesn't work well (fixed?)
Stop using essential matrix for pose estimation once you have a track (done)
Add kinematic model (done)
Run g2o to only optimize the latest pose (done)
Add search by projection to refind old map points (done)
Check if points are in the field of view of the camera (done)
BUGFIX: Fix moving up! (y axis flipped? nah, it's okay)
Add loading and saving of map support

TODO

Investigate if we need KeyFrames!
BUGFIX: Improve lockups to happen less
Improve init to not need REVERSE environment variable
Add optimizer for F
Add multiscale feature extractor
Add Linux libraries and OS check
Profile and speed up more (tomorrow 6/1/18)
Profile with flame!
Search by projection less stupidly
Don't add all points to the optimization graph for pose at least
Add automated test for freiburg running on commit
Check accuracy with ground truth

LICENSE

All my code is MIT licensed. Videos and libraries follow their respective licenses.

shrimo/twitchslam