uncertainty_qe

Experiments to address uncertainty in Quality Estimation

Requirements

Python 2.7
Numpy
Matplotlib
scikit-learn (for SVMs)
GPy (warped_gp_fixes branch in beckdaniel's fork)

Data

3 datasets:

WMT14 English-Spanish
EAMT11 English-Spanish
EAMT11 French-English

Features are the 17 baseline Quest features. Response variables are post-editing time per word (check Graham(2015), ACL).

Experiments sketch

For each dataset, we perform experiments using the following ML models:

SVM + Bagging
GP RBF Kernel Isotropic
GP RBF Kernel ARD
GP Matern32 Kernel ARD
Warped GP Matern32 Kernel ARD

Intrinsic evaluation is made using the following metrics:

NLPD (only available for GPs)
MAE
MSE
Pearson's correlation measure (check Graham(2015), ACL)

Extrinsic evaluation is made using the following tasks:

Reject option setting
Active learning setting
?

Everything is done via cross-validation (5 folds as default but this can be changed).

Reject options

Main idea: ignore predicitons with high uncertainty (variance). We plot curves measuring intrinsic metrics (Pearson's?) according to the top N% most confident predictions. Ideal curves should be monotonic.

Active learning