Got very different result between process_image.py and chester-xray web with same image.
Typepluto opened this issue · 5 comments
process_image.py result:
{'preds': {'Atelectasis': 0.5126234,
'Cardiomegaly': 0.06345778,
'Consolidation': 0.5328131,
'Edema': 0.2958934,
'Effusion': 0.53310466,
'Emphysema': 0.50264,
'Enlarged Cardiomediastinum': 0.5229955,
'Fibrosis': 0.5155035,
'Fracture': 0.51720035,
'Hernia': 0.0015778106,
'Infiltration': 0.5470137,
'Lung Lesion': 0.16751169,
'Lung Opacity': 0.8801794,
'Mass': 0.65471303,
'Nodule': 0.5540755,
'Pleural_Thickening': 0.50598806,
'Pneumonia': 0.033414263, <------------------------------------------------------------------------------
'Pneumothorax': 0.50849223}}
chester-xray web result:
Atelectasis,pred:0.16236889362335205,OP_POINT:0.07422872->normalized:0.5476036444030495
Consolidation,pred:0.14904548227787018,OP_POINT:0.038290843->normalized:0.5575821902452106
Infiltration,pred:0.18629415333271027,OP_POINT:0.09814756->normalized:0.5488697426669435
Pneumothorax,pred:0.03416414186358452,OP_POINT:0.0098118475->normalized:0.5122968015230744
Edema,pred:0.1063307672739029,OP_POINT:0.023601074->normalized:0.5423646990338368
Emphysema,pred:0.01815347746014595,OP_POINT:0.0022490358->normalized:0.507970145973699
Fibrosis,pred:0.02496715821325779,OP_POINT:0.010060724->normalized:0.5075289639347826
Effusion,pred:0.1678045243024826,OP_POINT:0.103246614->normalized:0.5359953535221347
Pneumonia,pred:0.2875974476337433,OP_POINT:0.056810737->normalized:0.8090469355374046 <------------------------
Pleural Thickening,pred:0.039720576256513596,OP_POINT:0.026791653->normalized:0.5066424231236653
Cardiomegaly,pred:0.01207550149410963,OP_POINT:0.050318155->normalized:0.11999149704624136
Nodule,pred:0.13887496292591095,OP_POINT:0.023985857->normalized:0.558856271064256
Mass,pred:0.24335533380508423,OP_POINT:0.01939503->normalized:0.7984534567199927
Hernia,pred:0.00022874104615766555,OP_POINT:0.042889766->normalized:0.0026666156928632527
Lung Lesion,pred:0.020575225353240967,OP_POINT:0.053369623->normalized:0.19276157668605762
Fracture,pred:0.06660972535610199,OP_POINT:0.035975814->normalized:0.5158885595408194
Lung Opacity,pred:0.7050759196281433,OP_POINT:0.20204692->normalized:1
Enlarged Cardiomedia.,pred:0.08681041747331619,OP_POINT:0.05015312->normalized:0.5192964246370511
The result of Pneumonia is very different. Is this a normal behavior?
And what's the difference of web and process_image.py?
The main difference is with the preprocessing and resizing because one uses skimage and the other uses built in web browser image resizing. The weights of the models are the same. We did a study in the original chester paper and only found small changes.
To isolate if it is the resizing issue try to scale and crop thr image to 224x224 and then try it wih both methods.
Does this happen for all the images you try? Can you share the image you used?
Sorry for the late reply.
Does this happen for all the images you try?
->Not as critical as this one. But other images still have some difference.
Can you share the image you used?
->I can send over the image via email, and please provide the email address for further sharing.
Did resizing the image fix the issue though? I think this is the main reason for differences.
My email is joseph at josephpcohen.com
Ok I figured it out. I forgot I added a scaling only in the web version so it would be more useful in predicting lung opacity and other targets. It scales the upper predictions (0.6 >) which made it easier to interpret the results because physicians didn't see 0.75 as a strong prediction when this is a strong prediction given this model.
https://github.com/mlmed/chester-xray/blob/master/res/js/system.js#L638
The XRV all model has a 1.3 scale: https://github.com/mlmed/chester-xray/blob/master/models/xrv-all-45rot15trans15scale/config.json
Let me know if this issue is not resolved. I'll close it for now.