Estimating 3D Motion & Forces

Zongmian Li, Jiri Sedlar, Justin Carpentier, Ivan Laptev, Nicolas Mansard and Josef Sivic

[Project Page] [arXiv] [Oral Slides] [Poster]

Description

This repository hosts the estimating stage implementation of the paper: Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video (Best Paper Finalist @ CVPR 2019).

As a quick setup, users can compile the trajectory estimator and evaluate its performance by running the demo script using precomputed 2D measurements.

If you are applying the method to new videos, you'll need the recognition stage to generate 2D measurements from pixels. In that case, consider installing the following open-source projects:

Human 2D poses: use Openpose.
Human 3D poses (required for trajectory initialization): any SMPL(-X) based pose estimator should be fine. For example, use HMR or FrankMocap.
Contact states: we used contact-recognizer. The hand-object detector is a nice alternative for it's trained on large-scale Internet data and should generalize well to novel scenes.
Object 2D keypoints: use object endpoint recognizer or any object segmentation model of your choice.

Consider ⭐ this repo if you find the work relevant. Feel free to open an issue if you need technical support or have suggestions.

Installation

First of all, note that we need CMake to generate native build files for Ceres and our motion-force estimator. CMake can be install via APT:

sudo apt install cmake

Use brew install cmake if you work on macOS.

Step I. Install Ceres

You can either download a stable release (Ceres 2.2 or later) from the official website or clone the latest version:

git clone https://ceres-solver.googlesource.com/ceres-solver
cd ceres-solver

Follow the instructions in Ceres installation guide to install dependencies, build and install Ceres. Make sure that Ceres can be found by CMake after make install.

Step II. Build the motion-force estimator

First, clone this repo and its submodules (using --recursive):

git clone --recursive https://github.com/zongmianli/Estimating-3D-Motion-Forces.git
cd Estimating-3D-Motion-Forces

Create a conda enviroment, using:
```
conda env create -f enviroment.yml
conda activate estmf
```
Alternatively, you can try the following steps if conda is too slow:
```
conda create --name estmf python=3.8
conda activate estmf
```
Install Pinocchio and the other dependencies:
```
conda install pinocchio -c conda-forge
pip install -r requirements.txt
```
Note that EigenPy should be available once Pinocchio installed. Install it manually if it's not the case, using conda install eigenpy -c conda-forge. EigenPy helps to bind our C++ solver in Python.
Generate build files using CMake. In /Estimating-3D-Motion-Forces, type:
```
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
```
Make sure to add -DCMAKE_BUILD_TYPE=Release at this stage. Otherwise Ceres will run in debug mode and will be very slow.
Finally, build the solver for our motion-force estimator:
```
make -j4
```
The process can take a few minutes. Finally you will find the compiled solver in /Estimating-3D-Motion-Forces/lib/solver.so. It's callable from Python.

Example usage

We provide a demo code showing how to use the code. In /Estimating-3D-Motion-Forces, run:

source scripts/demo.sh

This sample sequence is extracted from the Parkour dataset.

Citation

If you find this work helpful, please consider citing:

@InProceedings{li2019motionforcesfromvideo,
  author={Zongmian Li and Jiri Sedlar and Justin Carpentier and Ivan Laptev and Nicolas Mansard and Josef Sivic},
  title={Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video},
  booktitle={Computer Vision and Pattern Recognition (CVPR)},
  year={2019}
}

@article{li2022estimating,         
  title={Estimating 3D Motion and Forces of Human-Object Interactions from Internet Videos},
  author={Zongmian Li and Jiri Sedlar and Justin Carpentier and Ivan Laptev and Nicolas Mansard and Josef Sivic},
  journal={International Journal of Computer Vision},
  volume={130},
  number={2},
  pages={363--383},
  year={2022},
  publisher={Springer}
}