Question Regarding Model Results in ADEPT Paper
Closed this issue · 2 comments
Hi ADEPT creators,
I have a question regarding the ADEPT dataset you published in this repo and the accompanying paper. Specifically, I am interested in the modeling results presented in Table 5 (p. 7123) listing 3-Class Dev. Accuracy and 5-Class Dev. Accuracy.
Here, I am afraid I am uncertain whether the mentioned Dev set corresponds to the validation or test split you referenced in Table 4 (p. 7122) and devise accordingly as val.json and test.json through the linked Github - would you let me know which split/setting was used in the context of the paper? From a reader's perspective, this could be understood as one of the following options:
- optimization: dev set, evaluation: test set, with test set results reported in the paper
- optimization: test set, evaluation: dev set, with dev set results reported in the paper
- optimization: dev set, with dev set results reported in the paper
- a different setting
Thanks for your reponse!
Hi Ali,
Thank you very much for the quick answer and the clarification regarding the training/evaluation setup with the take-away that for the modeling approaches reported in the paper, the dev set was considered only (while the devised test set is not considered for the reported results in any way).
Best,
Anne