Data and code for ACL 2023 paper "RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations"
We officially support python 3.9. You could use following commands to install the required packages
pip install -r requirements.txt
We have released our dataset on HuggingFace. Use the following command to load the dataset (we use RobuT-WTQ as an example):
datasets = load_dataset("yilunzhao/robut", split="wtq")The dataset can also be found in robut_data.zip file.
To run each model on each RobuT subset, please execute the inference scripts in the inference_scripts directory. For example, use the following command to evaluate the performance of TAPEX on RobuT-WTQ:
bash inference_scripts/wtq/tapex.sh
The corresponding model output and scores can be found at outputs/wtq/tapex-preds.json and outputs/wtq/tapex-scores.json, respectively.
For any issues or questions, kindly email us at: Yilun Zhao (yilun.zhao@yale.edu).
@inproceedings{zhao-etal-2023-robut,
title = "{R}obu{T}: A Systematic Study of Table {QA} Robustness Against Human-Annotated Adversarial Perturbations",
author = "Zhao, Yilun and
Zhao, Chen and
Nan, Linyong and
Qi, Zhenting and
Zhang, Wenlin and
Tang, Xiangru and
Mi, Boyu and
Radev, Dragomir",
booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.acl-long.334",
doi = "10.18653/v1/2023.acl-long.334",
pages = "6064--6081",
}