CSV files required by illumination correction pipelines
hellowangqian opened this issue · 16 comments
Thanks a lot for sharing the code. I'm wondering if you can share more details of the illumination correction pipelines, and particularly it'll be much appreciated if you can provide the "loaddata.csv" and "LoadDataFile.csv" used in the pipelines. Thanks again and looking forward to your response.
Hi @hellowangqian I believe these files are very standard and in standard format. We can't provide the exact files as they include mostly paths to our internal folder structure. However we can provide a template in the new year if that will be helpful!
Thanks for your kind reply. A template with sensitive info removed would be very helpful.
The other thing I'd like to ask is line298 in Eval_BBBC021.py, why dropping those rows (17 18 19 20 21 22 60 61 62 63 64) when calculating NSCB?
@hellowangqian In the BBBC021 dataset there a some compounds/treatments which appear only in one batch hence we drop them from the NSCB evaluation as they have no matching MoA in a different batch - those are the rows in that sheet which correspond to those treatments.
TVN is done by channel on the image level. Features are made for each channel then concatenated for the final evaluation
Okay, hard to know exactly from the information you have given. the .csv file is just BBBC021_annotated.csv for both. It won't be because of 1 GPU. Are you using BBBC021 and the same labels?
Have you tried using the strong label (MoA/target - i.e. the same label you use in evaluating NSC/NSCB matching)? We found that this would very effectively and consistently push the NSC/NSCB close to 99% or even 100% within 50-100 epochs of training. If you can't reproduce that I would say there is a more fundamental problem with your implementation
Also check our training logs and see how yours compare. We found that aggregating the channels usually always resulted in superior NSC and NSCB scores compared to the individual channels e.g. https://github.com/crosszamirski/WS-DINO/blob/main/training_logs/NSC_NSCB_Weak_Compound.csv
Finally, if you are using our scripts in Eval_BBBC021.py and TSNE.py, they are quite hand crafted to match the format of our data sheets. They are not universal functions so be careful using them out the box!
It sounds like it might be an aggregation/normalisation issue. How we did it will be in TSNE.py but I think we concatenated the features before any normalisation (L2) prior to evaluation with NSC/NSCB matching
Thank you very much for clarifying things. I'm using BBBC021 and the same labels (i.e. those from this repository). Another thing that could vary is the illumination corrected images. Although I also used the cellprofiler pipeline files from this repository to get the illumination corrected images, I'm not sure if I have done it correctly. I got the csv file (consisting of 13200 rows for the whole list of images in BBBC021) from elsewhere to replace the missing "loaddata.csv " required by "BBBC021_IlluminationCalculate.cppipe" and a similar one (with added columns for specifying Illum.npy paths) for "BBBC021_IlluminationApply.cppipe".
Thanks for the suggestion of using the strong label, I'm trying it a.t.m.
I did check the training logs. My losses decrease slower than yours in general and end up with higher loss values for the same epochs.
I have also tried TSNE.py to combine three channels, no luck (same results as my own implementation of channel combination).
Hmm yeah does look like illumination correction could be a problem. We used a standard method as far as I am aware. let me know how you get on with MOA as the label. Also have a look at the self-attention maps, they could reveal a systematic illumination issue
Hello @crosszamirski , here is some additional information that may be helpful to identify what the issue could be.
- I did the experiment with MoA as the pseudo weak labels and the results of NSCB are as follows. Each row represents an epoch from 0, 10, 20, ... to 390 and four columns represent DAPI, Tubulin, Actin and Combined, respectively.
[0.7391304347826086, 0.8260869565217391, 0.6413043478260869, 0.8478260869565217,
0.6739130434782609, 0.8043478260869565, 0.6195652173913043, 0.7934782608695652,
0.717391304347826, 0.8152173913043478, 0.6413043478260869, 0.8043478260869565,
0.6739130434782609, 0.8260869565217391, 0.6521739130434783, 0.7934782608695652,
0.7065217391304348, 0.782608695652174, 0.6956521739130435, 0.7934782608695652,
0.7391304347826086, 0.8260869565217391, 0.7065217391304348, 0.8369565217391305,
0.782608695652174, 0.8586956521739131, 0.7391304347826086, 0.9456521739130435,
0.8043478260869565, 0.8913043478260869, 0.7608695652173914, 0.9565217391304348,
0.8152173913043478, 0.9239130434782609, 0.7608695652173914, 0.9565217391304348,
0.8152173913043478, 0.9130434782608695, 0.7717391304347826, 0.9891304347826086,
0.8152173913043478, 0.9130434782608695, 0.8152173913043478, 0.9891304347826086,
0.8260869565217391, 0.9239130434782609, 0.8152173913043478, 0.9891304347826086,
0.8260869565217391, 0.9456521739130435, 0.8152173913043478, 0.9891304347826086,
0.8260869565217391, 0.9347826086956522, 0.8260869565217391, 0.9891304347826086,
0.8260869565217391, 0.9456521739130435, 0.8369565217391305, 0.9891304347826086,
0.8152173913043478, 0.9565217391304348, 0.8260869565217391, 0.9891304347826086,
0.8260869565217391, 0.9456521739130435, 0.8369565217391305, 0.9891304347826086,
0.8152173913043478, 0.9021739130434783, 0.8369565217391305, 0.9891304347826086,
0.8152173913043478, 0.9347826086956522, 0.8369565217391305, 0.9891304347826086,
0.8043478260869565, 0.9130434782608695, 0.8478260869565217, 0.9891304347826086,
0.8152173913043478, 0.8913043478260869, 0.8478260869565217, 0.9891304347826086,
0.8260869565217391, 0.8913043478260869, 0.8478260869565217, 0.9891304347826086,
0.8369565217391305, 0.9021739130434783, 0.8478260869565217, 0.9891304347826086,
0.8152173913043478, 0.9021739130434783, 0.8586956521739131, 0.9891304347826086,
0.8369565217391305, 0.8695652173913043, 0.8369565217391305, 0.9891304347826086,
0.8260869565217391, 0.8913043478260869, 0.8369565217391305, 0.9891304347826086,
0.8369565217391305, 0.9021739130434783, 0.8586956521739131, 0.9891304347826086,
0.8260869565217391, 0.9130434782608695, 0.8260869565217391, 0.9891304347826086,
0.8260869565217391, 0.9347826086956522, 0.8369565217391305, 0.9891304347826086,
0.8369565217391305, 0.9021739130434783, 0.8478260869565217, 0.9891304347826086,
0.8478260869565217, 0.9130434782608695, 0.8260869565217391, 0.9891304347826086,
0.8260869565217391, 0.8913043478260869, 0.8260869565217391, 0.9891304347826086,
0.8369565217391305, 0.9021739130434783, 0.8260869565217391, 0.9891304347826086,
0.8478260869565217, 0.9021739130434783, 0.8260869565217391, 0.9891304347826086,
0.8478260869565217, 0.9239130434782609, 0.8260869565217391, 0.9891304347826086,
0.8478260869565217, 0.9239130434782609, 0.8260869565217391, 0.9891304347826086,
0.8369565217391305, 0.9239130434782609, 0.8260869565217391, 0.9891304347826086,
0.8478260869565217, 0.8913043478260869, 0.8260869565217391, 0.9891304347826086,
0.8369565217391305, 0.8913043478260869, 0.8260869565217391, 0.9891304347826086,
0.8478260869565217, 0.8913043478260869, 0.8369565217391305, 0.9891304347826086]
- For compound weak label experiments, the results of NSCB are as follows. Each row represents an epoch from 0, 10, 20, ... to 190 and four columns represent DAPI, Tubulin, Actin and Combined, respectively.
[0.7391304347826086, 0.8043478260869565, 0.6304347826086957, 0.8478260869565217,
0.6521739130434783, 0.8043478260869565, 0.6413043478260869, 0.75,
0.6086956521739131, 0.75, 0.6086956521739131, 0.6847826086956522,
0.6086956521739131, 0.8260869565217391, 0.5760869565217391, 0.6956521739130435,
0.6086956521739131, 0.7934782608695652, 0.6195652173913043, 0.782608695652174,
0.6304347826086957, 0.8043478260869565, 0.6195652173913043, 0.8369565217391305,
0.6413043478260869, 0.8804347826086957, 0.6521739130434783, 0.8478260869565217,
0.6521739130434783, 0.9021739130434783, 0.6739130434782609, 0.8369565217391305,
0.6630434782608695, 0.8695652173913043, 0.717391304347826, 0.8478260869565217,
0.6630434782608695, 0.9130434782608695, 0.7065217391304348, 0.8478260869565217,
0.6956521739130435, 0.8913043478260869, 0.717391304347826, 0.8804347826086957,
0.7065217391304348, 0.8913043478260869, 0.7065217391304348, 0.8913043478260869,
0.6956521739130435, 0.8804347826086957, 0.7065217391304348, 0.8804347826086957,
0.7065217391304348, 0.8695652173913043, 0.7282608695652174, 0.8586956521739131,
0.7065217391304348, 0.8804347826086957, 0.7282608695652174, 0.8478260869565217,
0.7065217391304348, 0.8586956521739131, 0.717391304347826, 0.8695652173913043,
0.7065217391304348, 0.8695652173913043, 0.717391304347826, 0.8804347826086957,
0.717391304347826, 0.8586956521739131, 0.7282608695652174, 0.8695652173913043,
0.7065217391304348, 0.8804347826086957, 0.717391304347826, 0.8913043478260869,
0.717391304347826, 0.8804347826086957, 0.717391304347826, 0.8913043478260869]
-
Some examples of attention map visualisation using compounds as the weak labels. From top to bottom: DAPI image, DAPI attention, Tubulin image, Tubulin attention, Actin image, Actin attention.
/7327614/210293582-bc4dab9d-fe7a-4707-9b7e-43da89502a1f.png) -
Some observations:
The combined results are not necessarily better than the best individual channel (usually the Tubulin channel) in my reproduced results;
In the compound weak label experiments, the individual channel results seem better than your reported results for Tubulin and Actin but worse for DAPI. And the 3 channel combined results are much worse than your reported ones.
Some thoughts:
I still suspect the issue is in the illumination correction step since I basically use your code for other steps. It would be great to have your "loaddata.csv" and "LoadDataFile.csv" at your convenience for validation.
Many thanks.
nice work, cheers. It's only a small difference from our results which is good. If we used a different ilum correction that could explain it, I'm sure these things are quite sensitive to preprocessing steps. You could also just try re-running and see if you get the same (our were pretty consistent but worth checking).
did you just train for 190 epochs? I think our best model was 200 epochs but I would probably train for more than 190 to be sure.
Will try and get that loaddata file to you asap
Hi @crosszamirski , thanks a million for your prompt reply. I understand the results for MoA weak labels are close to yours, however, the compound-as-weak-label results are far from satisfication. I will re-run this experiment with 400 epochs and update the results here though I doubt that will make a big difference.
It would also be helpful if you can provide some exemplar files for validating the illumination correction results, e.g., a pair of images before and after applying the correction, and the corresponding "xxxIllum_xxx.npy" file.
Yes going to need some time to get back to you on that so if you can find standard ilum correction elsewhere then try that. One more thing to help understand your results is plot tSNE maps, might reveal some insights
A quick question: when you do the illumination correction, do you use all images in BBBC021 for calculating the Illumination Correction Functions or only the MoA-labelled subset of BBBC021?
All of BBBC021. I have found our loaddata .csv files and they have the following headings: TableNumber ImageNumber Image_FileName_DAPI Image_PathName_DAPI Image_FileName_Tubulin Image_PathName_Tubulin Image_FileName_Actin Image_PathName_Actin Metadata_Plate Metadata_Well Image_FileName_DAPIIllum Image_FileName_ActinIllum Image_FileName_TubulinIllum