CDEvalSumm: Cross-Dataset Evaluation for Summarization

Descriptions and metrics code for EMNLP2020 findings paper:

CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems

(Yiran Chen*, Pengfei Liu*, Ming Zhong, Zi-Yi Dou, Danqing Wang, Xipeng Qiu, Xuanjing Huang)

Motivation

Many work evaluate summarization systems on in-domain setting (the model is trained and tested on the same dataset). In this work we try to understand model performance on different perspectives on a cross-dataset setting. The picture blow represents the main motivation (summarization systems get different rankings when evaluated under different measures where abstractive models are red while extractive ones are blue):

Two Research Questions

Q1: How do different neural architectures of summarizers influence the cross-dataset generalization performances?
Q2: Do different generation ways (extractive and abstractive) of summarizers influence the cross-dataset generalization ability?

Evaluation Systems

Extractive summarizers: $LSTM_{non}, Trans_{non}, Trans_{auto}, BERT_{non}, BERT_{match}$

Abstractive summarizers: $L2L_{ptr}^{cov}, L2L_{ptr}, L2L, T2T, BE2T, BART$

Systems		Paper	Bib
Abs-Sum	LSTM_{non}	Content Selection in Deep Learning Models of Summarization	Bib
	Trans_non	Text Summarization with Pretrained Encoders	Bib
	Trans_{auto}	Searching for Effective Neural Extractive Summarization: What works and What’s Next	Bib
	BERT_{non}	Text Summarization with Pretrained Encoders	Bib
	BERT_{match}	Extractive Summarization as Text Matching	Bib
Ext-Sum	L2L^{cov}_{ptr}	Get to the point: Summarization with Pointer-Generator Networks	Bib
	L2L_{ptr}	Get to the point: Summarization withpointer-generator networks	Bib
	L2L	CDEvalSumm: An Empirical Study of Cross-Dataset Evaluationfor Neural Summarization Systems	Bib
	T2T	Text Summarization with Pretrained Encoders	Bib
	BE2T	Text Summarization with Pretrained Encoders	Bib
	BART	Bart: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension	Bib

Datasets

Datasets
- CNN/Dailymail
- Xsum
- Pubmed
- Bigpatent B
- Reddit TIFU

Evaluation Metrics

Semantic Equivalenc (ROUGE)
Factuality (Factcc)
Dataset bias (Detailed explanation is displayed in our paper and the code can refer to Data-bias-metrics/)
- Coverage
- Copy length
- Novelty
- Repetition
- Sentence fusion score

Cross-dataset Measures

Stiffness

$r^{\mu} = \frac{1}{N\times N}\sum_{i,j} {\mathbf{U}}_{ij}$
${\mathbf{U}}_{ij}$ : the metric score when model is trained on dataset i and tested on dataset j.
Stableness

$r^{\sigma} = \frac{1}{N\times N}\sum_{i,j} \mathbf{U}_{ij}/ \mathbf{U}_{jj}\times100 \%$
${\mathbf{U}}_{ij}$ : the metric score when model is trained on dataset i and tested on dataset j.

Experiment Results

The stiffness and stableness of various summarizers are displayed below. For fine-grained results and comprehensive analysis please refer to the paper.

zide05/CDEvalSumm