Question Regarding Model Results in ADEPT Paper

[AnneroseEichel]

Hi ADEPT creators,

I have a question regarding the ADEPT dataset you published in this repo and the accompanying paper. Specifically, I am interested in the modeling results presented in Table 5 (p. 7123) listing 3-Class Dev. Accuracy and 5-Class Dev. Accuracy.

Here, I am afraid I am uncertain whether the mentioned Dev set corresponds to the validation or test split you referenced in Table 4 (p. 7122) and devise accordingly as val.json and test.json through the linked Github - would you let me know which split/setting was used in the context of the paper? From a reader's perspective, this could be understood as one of the following options:

1. optimization: dev set, evaluation: test set, with test set results reported in the paper
2. optimization: test set, evaluation: dev set, with dev set results reported in the paper
3. optimization: dev set, with dev set results reported in the paper
4. a different setting

Thanks for your reponse!

[aemami1]

Hi there, Thank you so much for your great question, and for the clarity in explaining your problem. The configuration used is as follows: optimization was conducted on the development set. Specifically, training and tuning were performed on the training set, while the validation set was used to approximate the generalization of the results. The results in table 5 correspond to the validation set, and for that matter, any time we say "dev set" or "development set" in our results, we are referring to the validation set performance – we should have been more specific here, our apologies! Thus, your third option aligns with our approach: "optimization: dev set, with dev set results reported in the paper." Does this help clarify? Best, Ali

…

------------------------------------------------- Ali Emami, Ph.D. Assistant Professor Department of Computer Science Brock University Email: ***@***.*** Web: http://www.cosc.brocku.ca/~aemami/

________________________________ From: Anne ***@***.***> Sent: Friday, July 12, 2024 9:34 AM To: aemami1/adept ***@***.***> Cc: Subscribed ***@***.***> Subject: [aemami1/adept] Question Regarding Model Results in ADEPT Paper (Issue #2) Hi ADEPT creators, I have a question regarding the ADEPT dataset you published in this repo and the accompanying paper. Specifically, I am interested in the modeling results presented in Table 5 (p. 7123) listing 3-Class Dev. Accuracy and 5-Class Dev. Accuracy. Here, I am afraid I am uncertain whether the mentioned Dev set corresponds to the validation or test split you referenced in Table 4 (p. 7122) and devise accordingly as val.json and test.json through the linked Github - would you let me know which split/setting was used in the context of the paper? From a reader's perspective, this could be understood as one of the following options: 1. optimization: dev set, evaluation: test set, with test set results reported in the paper 2. optimization: test set, evaluation: dev set, with dev set results reported in the paper 3. optimization: dev set, with dev set results reported in the paper 4. a different setting Thanks for your reponse! — Reply to this email directly, view it on GitHub<#2>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AH4QMM4RYDASF4LZPE4KBLLZL7LNLAVCNFSM6AAAAABKY6HLL6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGQYDKNRTGE3DCMQ>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[AnneroseEichel]

Hi Ali,

Thank you very much for the quick answer and the clarification regarding the training/evaluation setup with the take-away that for the modeling approaches reported in the paper, the dev set was considered only (while the devised test set is not considered for the reported results in any way).

Best,
Anne