/SGGpoint

[CVPR 2021] Exploiting Edge-Oriented Reasoning for 3D Point-based Scene Graph Analysis

Primary LanguageJupyter Notebook

SGGpoint

This repository contains a PyTorch implementation of the following paper:

Exploiting Edge-Oriented Reasoning for 3D Point-based Scene Graph Analysis

Chaoyi Zhang, Jianhui Yu, Yang Song, Weidong Cai

CVPR 2021

Abstract Scene understanding is a critical problem in computer vision. In this paper, we propose a 3D point-based scene graph generation (SGGpoint) framework to effectively bridge perception and reasoning to achieve scene understanding via three sequential stages, namely scene graph construction, reasoning, and inference. Within the reasoning stage, an EDGE-oriented Graph Convolutional Network (EdgeGCN) is created to exploit multi-dimensional edge features for explicit relationship modeling, together with the exploration of two associated twinning interaction mechanisms between nodes and edges for the independent evolution of scene graph representations. Overall, our integrated SGGpoint framework is established to seek and infer scene structures of interest from both real-world and synthetic 3D point-based scenes. Our experimental results show promising edge-oriented reasoning effects on scene graph generation studies. We also demonstrate our method advantage on several traditional graph representation learning benchmark datasets, including the node-wise classification on citation networks and whole-graph recognition problems for molecular analysis.

[Project page] [Arxiv] [Supp.] [Video]

img

Figure. Our proposed 3D point-based scene graph generation (SGGpoint) framework consisting of three sequential stages, namely, scene graph construction, reasoning, and inference.

Dataset

A quick glance at some features of our cleaned 3DSSG-O27R16 dataset (compared to the original 3DSSG dataset):

  • dense point cloud representation with color and normal vector info. encoded - see Sec. A - Point Cloud Sampling;
  • with same scene-level split applied on 3DSSG - but with FullScenes (i.e., original graphs) instead of SubScenes (subgraphs of 4-9 nodes in 3DSSG);
  • with small / partial scenes of low quality excluded - see this list (officially announced in 3DSSG's FAQ Page);
  • with object-level class imbalance alleviated - see Sec. B1 - Node (object) Remapping;
  • with edge-wise comparative relationships (e.g., more-comfortable-than) filtered out - we focus on structural relationships instead;
  • reformulate the edge predictions from a multi-label classification problem to a multi-class one - see Sec. B2 - Edge (Relationship) Relabelling;

To obtain our preprocessed 3DSSG-O27R16 dataset, please follow the instructions in our project page - or you could also derive these preprocessed data yourselves by following this step-by-step preprocessing guidance with scripts provided.

Code

This repo. also contains Pytorch implementation of the following modules:

code coming soon, stay tuned!

Citation

If you find our data or project useful in your research, please cite:

@inproceedings{SGGpoint,
    title = {Exploiting Edge-Oriented Reasoning for 3D Point-based Scene Graph Analysis},
    author = {Zhang, Chaoyi and Yu, Jianhui and Song, Yang and Cai, Weidong},
    booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2021}
}

Acknowledgments

This repo. template was borrowed from Yang Zhou's Project.