ComPhy
This repository holds the code for the paper.
ComPhy: Compositional Physical Reasoning ofObjects and Events from Videos, Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B. Tenenbaum, Chuang Gan, (Under review)
Framework
Code Preparation
git clone https://github.com/zfchenUnique/executor_comphy.git
Installation
pip install -r requirements
Data Preparation
- Download videos, video annotation, questions from the official website.
Fast Evaluation
- Download the regional proposals with attribute and physical property prediction from Google drive
- Download the dynamic predictions from Google drive
- Run executor for factual questions.
sh scripts/test_oe_release.sh
- Run executor for multiple-choice questions.
sh scripts/test_mc_release.sh
Supporting sub-modules
Physical Property Learner and Dynamic predictor
Please refer to this repo for property learning and dynamics prediction.
Perception
This module uses the NS-VQA's perception module object detection and visual attribute extraction.
Program parser
This module uses the NS-VQA's program parser module to tranform language into executable programs.