/HAKE-Action-Torch

HAKE-Action in PyTorch

Apache License 2.0Apache-2.0

HAKE-Action-Torch

Seven-in-One: CVPR'18 (Part States), CVPR'19 (interactiveness), CVPR'20 (PaStaNet, Dj-RN, SymNet), NeurIPS'20 (IDN), TPAMI(Extended TIN).

HAKE-Action-Torch (PyTorch) is a project to open the SOTA action understanding studies based on our project: Human Activity Knowledge Engine. It includes SOTA models and their corresponding HAKE-enhanced versions based on our six papers (CVPR'18/19/20, NeurIPS'20). The TensorFlow version of HAKE-Action is here.

Currently, it is manintained by Yong-Lu Li, Xinpeng Liu and Zhanke Zhou, Hongwei Fan.

News: (2021.2.7) Upgraded HAKE-Activity2Vec is released! Images/Videos --> human box + ID + skeleton + part states + action + representation. [Description]

(2021.1.15) Our extended version of TIN (Transferable Interactiveness Network) is accepted by TPAMI!

(2020.10.27) The code of IDN (Paper) in NeurIPS'20 is released!

Project

HAKE-Action-Torch
  ├──Master Branch                          # Unified pipeline; CVPR'18/20, PaStanet and Part States.
  ├──IDN-(Integrating-Decomposing-Network)  # NeurIPS'20, HOI Analysis: Integrating and Decomposing Human-Object Interaction.
  ├──DJ-RN-Torch                            # CVPR'20, Detailed 2D-3D Joint Representation for Human-Object Interaction.
  ├──TIN-Torch                              # CVPR'19, Transferable Interactiveness Knowledge for Human-Object Interaction Detection.
  └──SymNet-Torch                           # CVPR'20, Symmetry and Group in Attribute-Object Compositions.

Papers

Results on HICO-DET with different object detections.

Method Detector HAKE Full(def) Rare(def) None-Rare(def) Full(ko) Rare(ko) None-Rare(ko)
TIN COCO - 17.54 13.80 18.65 19.75 15.70 20.96
TIN COCO HAKE-HICO-DET 22.12 20.19 22.69 24.06 22.19 24.62
TIN COCO HAKE-Large 22.66 21.17 23.09 24.53 23.00 24.99
TIN-PAMI COCO - 20.93 18.95 21.32 23.02 20.96 23.42
DJ-RN COCO - 21.34 18.53 22.18 23.69 20.64 24.60
IDN COCO - 23.36 22.47 23.63 26.43 25.01 26.85
IDN COCO+HICO-DET - 26.29 22.61 27.39 28.24 24.47 29.37
TIN GT Pairs - 34.26 22.90 37.65 - - -
IDN GT Pairs - 43.98 40.27 45.09 - - -

Results on V-COCO.

As VCOCO is built on COCO, thus finetuning detector on VCOCO basically contributes marhinally to performance.

Method HAKE AP(role)
TIN - 47.8
TIN HAKE-Large 51.0
TIN-PAMI - 49.1
IDN - 53.3

Results on Ambiguous-HOI.

Method mAP
TIN 8.22
DJ-RN 10.37

Results on PaStaNet-HOI

Method mAP
TIN-PAMI 15.38

Modules

Unified Model

Coming soon.

Activity2Vec (CVPR'20)

The independent Torch version is in: Activity2Vec (A2V).

IDN (NeurIPS'20)

The independent Torch version is in: IDN.

DJ-RN (CVPR'20)

The independent Torch version is in: DJ-RN-Torch

TIN (CVPR'19)

The independent Torch version is in: TIN-Torch

SymNet (CVPR'20)

Coming soon.

Citation

If you find our works useful, please consider citing:

---IDN:
@inproceedings{li2020hoi,
  title={HOI Analysis: Integrating and Decomposing Human-Object Interaction},
  author={Li, Yong-Lu and Liu, Xinpeng and Wu, Xiaoqian and Li, Yizhuo and Lu, Cewu},
  booktitle={NeurIPS},
  year={2020}
}
---HAKE:
@inproceedings{li2020pastanet,
  title={PaStaNet: Toward Human Activity Knowledge Engine},
  author={Li, Yong-Lu and Xu, Liang and Liu, Xinpeng and Huang, Xijie and Xu, Yue and Wang, Shiyi and Fang, Hao-Shu and Ma, Ze and Chen, Mingyang and Lu, Cewu},
  booktitle={CVPR},
  year={2020}
}
@inproceedings{lu2018beyond,
  title={Beyond holistic object recognition: Enriching image understanding with part states},
  author={Lu, Cewu and Su, Hao and Li, Yonglu and Lu, Yongyi and Yi, Li and Tang, Chi-Keung and Guibas, Leonidas J},
  booktitle={CVPR},
  year={2018}
}
---DJ-RN
@inproceedings{li2020detailed,
  title={Detailed 2D-3D Joint Representation for Human-Object Interaction},
  author={Li, Yong-Lu and Liu, Xinpeng and Lu, Han and Wang, Shiyi and Liu, Junqi and Li, Jiefeng and Lu, Cewu},
  booktitle={CVPR},
  year={2020}
}
---TIN
@inproceedings{li2019transferable,
  title={Transferable Interactiveness Knowledge for Human-Object Interaction Detection},
  author={Li, Yong-Lu and Zhou, Siyuan and Huang, Xijie and Xu, Liang and Ma, Ze and Fang, Hao-Shu and Wang, Yanfeng and Lu, Cewu},
  booktitle={CVPR},
  year={2019}
}
---SymNet
@inproceedings{li2020symmetry,
  title={Symmetry and Group in Attribute-Object Compositions},
  author={Li, Yong-Lu and Xu, Yue and Mao, Xiaohan and Lu, Cewu},
  booktitle={CVPR},
  year={2020}
}

TODO

  • TIN-based element analysis
  • Refined Activity2Vec
  • Extended DJ-RN
  • SymNet in Torch
  • Unified model (better A2V, early/late fusion, new representation)

HAKE[website] is a new large-scale knowledge base and engine for human activity understanding. HAKE provides elaborate and abundant body part state labels for active human instances in a large scale of images and videos. With HAKE, we boost the action understanding performance on widely-used human activity benchmarks. Now we are still enlarging and enriching it, and looking forward to working with outstanding researchers around the world on its applications and further improvements. If you have any pieces of advice or interests, please feel free to contact Yong-Lu Li (yonglu_li@sjtu.edu.cn).

If you get any problems or if you find any bugs, don't hesitate to comment on GitHub or make a pull request!

HAKE-Action-Torch is freely available for free non-commercial use, and may be redistributed under these conditions. For commercial queries, please drop an e-mail. We will send the detail agreement to you.