Setup

Requirements

Install Python dependency

pip install -r requirements.txt

Install KenLM

./install_kenlm.sh

Download data and pretrained model

Download the zip file from Google Drive, and copy it to the folder of code.

unzip acl18confidence_data_model.zip

Precompute Confidence Metrics

Train a language model

./train_lm.sh ifttt
./train_lm.sh django

Precompute all the confidence metrics, and perform uncertainty backpropagation

(This step can be skipped. The precomputed data have been cached in the zip file.)

# arg2: gpu id
./compute_metric.sh ifttt 0
./compute_metric.sh django 0

The intermediate results are saved to "data_model/[ifttt|django]/*.eval".

Usage

Confidence Estimation

  • Train a confidence scoring model.
  • Compute spearman's rho between confidence scores and F1 for the full model and ablation models (w/o model uncertainty, w/o data uncertainty, and w/o input uncertainty).
./train_scoring.sh ifttt
./train_scoring.sh django

Uncertainty Interpretation

Evaluate confidence backpropagation and attention-based method against inferred ground truth.

./eval_conf_bp_att.sh ifttt
./eval_conf_bp_att.sh django

Acknowledgments