This provides the source code for our paper.
The datasets/ folder contains the data for the two datasets used in our paper - Deepfake and RIP.
The test_with_statistics.csv contains test data with several statistics computed from the textdescriptives library.
The RIP Dataset can also be found on HuggingFace Datasets.
This can be found in the libauc_training directory. This includes the TextDataset LibAUCTrainer, Experiment classes. For example usage, see an example Jupyter notebook that uses the Experiment class.
Experiment data - such as predictions and metrics - can be found in the respective folders. Jupyter (Kaggle) notebooks are also present. Models can be found in the latest GitHub release as assets.
The auc_scores.csv and ap_scores.csv file reports the respective metric based on the blocking factor (LLM model or LLM family). The factor column maps to the test set while individual columns correspond to the classifier with a training set containing all texts with that specific factor.
@misc{
thorat2024llmsdifficultdetectdetailed,
title={Which LLMs are Difficult to Detect? A Detailed Analysis of Potential Factors Contributing to Difficulties in LLM Text Detection},
author={Shantanu Thorat and Tianbao Yang},
year={2024},
eprint={2410.14875},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2410.14875},
}