Performance estimation

[grayfall]

Dear DeepSeqPan authors,

Thank you very much for developing and publishing your software. I have trained DeepSeqPan as per your recommendations without introducing any modifications to the training code and data, yet the performance I have witnessed thus far appears to be at odds with the results you show in your publication. Here are several performance metrics estimated on https://github.com/pcpLiu/DeepSeqPan/blob/master/dataset/weekly_data_all_rm_duplicate.txt

Unweighted accuracy: 0.49
F1-score: 0.39
Precision: 0.75
Recall: 0.26

Am I doing something wrong? I am looking forward to your reply.

Best Regards,
Ilia Korvigo.

[pcpLiu]

Hey @grayfall,

Seems you evaluated on all weekly benchmark samples together? Usually we evaluate model on each allele record in weekly benchmark dataset, separately.

You can find our performance and others' on each benchmark allele record in Table S1 at supporting material (which can be found here)

Also, since IEDB weekly benchmark calculates AUC and SRCC, we also used that metrics as you can find in table S1.