replicated results differ significantly from the original paper

[lcx0329]

Hello, our replicated results differ significantly from the original paper. Can you please provide a more specific replication process?

[wellido]

Hello, thanks for your interest in our work. Could you please let me know which model and dataset you use?

Following the example.py.

1. prepare your models (two models, with and without dropout layers). If necessary, I can send our used models to you
2. prepare two sets of data (with labels and without labels).
3. record the accuracy of the labeled data, see Line 14 in example.py.
4. use function drop_check_func to calculate the base accuracy map.
5. use function Aries_estimation to compute the accuracy of the unlabeled set.

[lcx0329]

I tried all the data sets of the models, but none of them reached the results given in the paper. If possible, please send me the models you use. My email is 178266098@qq.com Thank you!
I also want to ask, does the training model have a dropout layer and what parameters need to be changed?

[wellido]

Hi, sent it to you, you can try it. Please note that we only test our method using the dataset with severity=1. When the severity is large the data distribution is too far away from the original test data, Aries might not work well. For the model, you need to train one model first, initialize another model with a dropout layer, and then copy the weights of the trained model to the dropout model. The architecture is defined in the folder of models.