This repository contains code for the paper "Multi-Concept Learning for Scene Graph Generation". This code is based on Scene-Graph-Benchmark.pytorch.
Existing Unbiased Scene Graph Generation (USGG) methods only focus on addressing the predicate-level imbalance that high-frequency classes dominate predictions of rare ones, while overlooking the concept-level imbalance. Actually, even if predicates themselves are balanced, there is still a significant concept-imbalance within them due to the long-tailed distribution of contexts (i.e., subject-object combinations). This concept-level imbalance poses a more pervasive and challenging issue compared to the predicate-level imbalance since subject-object pairs are inherently complex in combinations. To address the issue, we propose Multi-Concept Learning (MCL), a novel concept-level balanced learning framework orthogonal to existing SGG methods. MCL first quantifies the concept-level imbalance across predicates in terms of different amounts of concepts, representing as multiple concept-prototypes within the same class. It then effectively learns concept-prototypes by applying the Concept Regularization (CR) technique. Furthermore, to achieve balanced learning over different concepts, we introduce the Concept-based Balanced Memory (CBM), which guides SGG models to generate balanced representations for concept-prototypes. Finally, we introduce a novel metric, mean Context Recall (mCR@K), as a complement to mean Recall (mR@K), to evaluate the model's performance across concepts (determined by contexts) within the same predicate. Extensive experiments demonstrate the remarkable efficacy of our model-agnostic strategy in enhancing the performance of benchmark models on both VG-SGG and OI-SGG datasets, leading to new state-of-the-art achievements in two key aspects: predicate-level unbiased relation recognition and concept-level compositional generability.
WThe Overview of our Multi-Concept Learning (MCL) framework. It contains three parts: Concept-Prototype Construction (CPC), Concept-based Balanced Memory (CBM), and Concept Regularization (CR). The CPC assesses the semantic scales of each predicate from the SGG dataset and then quantifies the semantic scales into different amounts of concept-prototypes within the same predicate. Subsequently, CBM produces the balanced relation representations across different concepts within predicates, enabling the SGG model to equally attend to all concepts. Finally, the CR alleviates the predicate-level semantic overlap while enhancing the discriminability among concept-prototypes.
Follow DATASET.md for instructions of dataset preprocessing.
Follow the instructions to install and use the code. Also, we provide scripts for training models with MCL our model (in scripts/885train_[motifs/penet].sh(https://github.com/XinyuLyu/G-USGG/blob/master/script))
The trained models(Motif-MCL) on Predcls\SGCLs\SGDet are released as below.
| Predcls | SGCLs | SGDet |
|---|---|---|
| Motif-MCL-Predcls | Motif-MCL-SGCLS | Motif-MCL-SGDet |
| PENET-MCL-Predcls | PENET-MCL-SGCLS | PENET-MCL-SGDet |
Be free to contact me (xinyulyu68@gmail.com) if you have any questions!
The code is implemented based on Scene-Graph-Benchmark.pytorch, and SGG-G2S. Thanks for their great works!
@inproceedings{sgg:MCL,
author = {Xinyu Lyu and
Lianli Gao and
Junlin Xie and
Pengpeng Zeng and
Yulu Tian and
Jie Shao and
Heng Tao Shen},
title = {Multi-Concept Learning for Scene Graph Generation},
booktitle = {CoRR},
year = {2023}
}