Big thanks to my awesome teammates @ewrfcas and @leolemon214. In this repository you will find the code for the 21th place solution, puzzlingly shaking from public LB 3th. Our team finished as 3/1233 with a micro F1-score of 0.71 on the public test set and 21/1233 with a micro F1-score of 0.67 on the private test set. The challenge was to predict short and long answer responses to real questions about Wikipedia articles.
The dataset is provided by Google's Natural Questions, but contains its own unique private test set.
# Get train set and public test set
sudo pip install --upgrade kaggle
mkdir .kaggle
# Replace "MYUSER" and "MYKEY" with your credentials. You can create them on:
# `https://www.kaggle.com` -> `My Account` -> `Create New API Token`
echo '{"username":"MYUSER","key":"MYKEY"}' > ~/.kaggle/kaggle.json
chmod 600 ~/.kaggle/kaggle.json
kaggle competitions download -c tensorflow2-question-answering
for f in *.zip; do unzip $f; done
rm *.zip
# Get dev set
wget https://storage.cloud.google.com/natural_questions/v1.0-simplified/nq-dev-all.jsonl.gz
gunzip *.gz
mkdir -p tf2qa/data
mv *.jsonl tf2qa/dataTF2-QA
├── LICENSE
├── README.md
├── input/
├── notebooks/
├── output/
├── refs/
├── tf2qa/ <- pipeline code base
│ ├── data/ <- data folder
│ ├── dataset/ <- preprocessed feature folder
│ ├── checkpoints/ <- saved model folder
│ ├── roberta_large/ <- roberta_large init folder, containing config, vocab, weight ...
│ ├── albert_xxlarge/ <- albert_xxlarge init folder, containing config, vocab, weight ...
└── visulization <- visualization code base# get the code
git clone https://github.com/mikelkl/TF2-QA.git
cd TF2-QA/tf2qa
# STEP1 将数据集去HTML化，然后按照字数阈值（600）划分存储
python build_splited_data.py
# STEP2 将分段后的语料选出top8
python build_tfidf.py
# STEP3 roberta_large LS preprocess
python roberta_preprocess.py
# STEP4 roberta_large LS train
python train_roberta_topk.py
# STEP5 albert short preprocess
python albert_short_preprocess.py
# STEP6 albert short train
python train_albert_short.py
# STEP7 ensemble pipeline
python pipline_roberta_albert.py| No | Technique | Pros | Cons | Effect |
|---|---|---|---|---|
| 1 | TF-IDF paragraph selection | Shorten doc resulting faster inference speed and better accuracy | May loss some context information | - dev f1 +1.8%, - public LB f1 -1% |
| 2 | Sample negative features till 1:1 | Balance pos and neg | Cause longer training time | dev f1 +2.248% |
| 3 | Multi-process preprocessing | Accelerate preprocessing, especially on training data | Require multi-core CPU | xN faster (with N processes) |
| No | Model Architecture | Idea | Performance |
|---|---|---|---|
| 1 | Roberta-Large joint with long/short span extractor | 1. Jointly model: - answer type - long span - short span 2. Output topk start/end logits/index |
dev f1 63.986% |
| 2 | Albert-xxlarge joint with short span extractor | Jointly model: - answer type - short span |
def short-f1 69.364% |
All of above model architectures were pretrained on SQuAD dataset by ourselves.
| No | Trick | Effect |
|---|---|---|
| 1 | If answer_type is yes/no, output yes/no rather than short span | public LB f1 +6% |
| 2 | 1. If answer_type is short, output long span and short span 2. If answer_type is long, output long span only 3. If answer_type is none, output neither long span nor short span |
public LB f1 +8% |
| 3 | Choose the best long/short answer pair from topk * topk kind of long/short answer combinations | dev f1 +0.435% |
| 4 | long_score = summary.long_span_score - summary.long_cls_score - summary.answer_type_logits[0]short_score = summary.short_span_score - summary.short_cls_score - summary.answer_type_logits[0] |
- dev f1 +2.12% - public LB +2% |
| 5 | Increase long [CLS] logits multiplier threshold to increase null long answer | dev long-f1 +3.491% |
| 6 | Decrease short answer_type logits divisor threshold to increase null short answer | dev short-f1 ? |
| No | Idea | Effect |
|---|---|---|
| 1 | For long answer, We vote long answers of 2 Roberta-Large joint with long/short span extractor models |
dev long-f1 +3.341% |
| 2 | For short answer, use step 1 result to locate predicted long answer candidate as input, We vote short answers of 2 Roberta-Large joint with long/short span extractor models and 4 Albert-xxlarge joint with short span extractor models |
- dev short-f1 +2.842% - dev f1 67.569%, +2.635% - public LB 71%, +5% - private LB 67% |