how to calculate uncertainty estimates

[Nero0113]

Following your steps, I divided cnews into two tag sets and trained two teacher models, but when I am ready to run monte_carlo.py by myself, I do not find the correct steps to proceed. Please, how can I train a weight file by myself, the steps you give in the web page are not enough to get the np file.

[TobiasLee]

Thanks for the question. If you want to prepare the uncertainty score files, you can use the teacher models to perform predictions on the whole dataset and save the prediction uncertainty to form a score file.
We uploaded the script used for running the inference 'scripts/md_cnews.sh' and the corresponding main program 'run_mc.py', you can check the models/monte_carlo.py for detailed computation.
If you have further questions, please do not hesitate to discuss them here, and we are glad to help.

[Nero0113]

Thank you for your kind help, best regard.

[Nero0113]

Thanks for the question. If you want to prepare the uncertainty score files, you can use the teacher models to perform predictions on the whole dataset and save the prediction uncertainty to form a score file. We uploaded the script used for running the inference 'scripts/md_cnews.sh' and the corresponding main program 'run_mc.py', you can check the models/monte_carlo.py for detailed computation. If you have further questions, please do not hesitate to discuss them here, and we are glad to help.

Please tell me how to add small perturbations in the inference process, I didn't get it from the code.

[TobiasLee]

The monte carlo dropout is easy to implement with the inner Dropout modules of teacher models.
We just make sure the teacher models are in the training mode(default setting) with Dropout activated, and run forward prediction several times to get the different predictions with different dropout masks:
https://github.com/lancopku/MUKI/blob/main/models/monte_carlo.py#L73