Example on Clinical and Slide table
Tato14 opened this issue · 5 comments
Hi! What a great resource, thanks for sharing!
I was wondering if you could specify a bit more on the structure of the Clinical Data table and slide Data table.
I guess the first should have something like:
ID;isMISH;stage
HE1;Yes,4
HE2;No,2
[...]
But I am not quite sure how the second table should be. A list of tiles with every row a pair of ID;path?
@Tato14 Thank you for using this repo. I would be happy if you communicate with me further which will help me to improve the code. About your question, you can find example clini and slide table in this repo : https://github.com/jnkather/DeepHistology .
@narminGhaffari, thanks for the feedback. Everything seems to work now.
I has to modify most of the paths (from \\ to /), since (I suppose), you were working with Windows.
I started a work with ~1300 WSI using resnet with default parametres. Here's the Report.txt:
**********************************************************************
Namespace(adressExp='/mnt/isilon/Lung_TCGA/train_Level1/ExperimentFile_resnet_default.txt', batch_size=128, clini_dir=['/mnt/isilon/Lung_TCGA/train_Level1/LungDX/LungDX_CLINI.xlsx'], csv_name='CLEANED_DATA', datadir_train=['/mnt/isilon/Lung_TCGA/train_Level1/LungDX/BLOCKS_NORM_MACENKO'], feat_dir=['/mnt/isilon/Lung_TCGA/train_Level1/LungDX/FEATURES'], freeze_Ratio=0.5, gpuNo=0, k=3, lr=0.0001, maxBlockNum=512, max_epochs=20, model_name='resnet', numHighScoreBlocks=20, numHighScorePatients=10, opt='adam', projectFolder='/mnt/isilon/Lung_TCGA/train_Level1/ExperimentFile_resnet_default_lung_type', project_name='ExperimentFile_resnet_default', reg=1e-05, seed=1, slide_dir=['/mnt/isilon/Lung_TCGA/train_Level1/LungDX/LungDX_SLIDE.csv'], target_label='lung_type', target_labels=['lung_type'], train_full=False, useClassicModel=True)
**********************************************************************
LOADING DATA FROM/mnt/isilon/Lung_TCGA/train_Level1/LungDX/BLOCKS_NORM_MACENKO...
No errors so far but I am a bit worried since the whole process is runing since yersterday, I am not able to find any process related to this and nvidia-smi looks empty. Is this normal? Or it just felt silently? Any idea? Thanks
Ok, I found the problem. It seems that the process finishes if project_folder is already present. It seems to be related with the ClassicTraining, concretely in the if statement. Seems that the continue is not well placed, I think
@narminGhaffari, thanks for the feedback. Everything seems to work now. I has to modify most of the paths (from
\\to/), since (I suppose), you were working with Windows. I started a work with ~1300 WSI using resnet with default parametres. Here's theReport.txt:********************************************************************** Namespace(adressExp='/mnt/isilon/Lung_TCGA/train_Level1/ExperimentFile_resnet_default.txt', batch_size=128, clini_dir=['/mnt/isilon/Lung_TCGA/train_Level1/LungDX/LungDX_CLINI.xlsx'], csv_name='CLEANED_DATA', datadir_train=['/mnt/isilon/Lung_TCGA/train_Level1/LungDX/BLOCKS_NORM_MACENKO'], feat_dir=['/mnt/isilon/Lung_TCGA/train_Level1/LungDX/FEATURES'], freeze_Ratio=0.5, gpuNo=0, k=3, lr=0.0001, maxBlockNum=512, max_epochs=20, model_name='resnet', numHighScoreBlocks=20, numHighScorePatients=10, opt='adam', projectFolder='/mnt/isilon/Lung_TCGA/train_Level1/ExperimentFile_resnet_default_lung_type', project_name='ExperimentFile_resnet_default', reg=1e-05, seed=1, slide_dir=['/mnt/isilon/Lung_TCGA/train_Level1/LungDX/LungDX_SLIDE.csv'], target_label='lung_type', target_labels=['lung_type'], train_full=False, useClassicModel=True) ********************************************************************** LOADING DATA FROM/mnt/isilon/Lung_TCGA/train_Level1/LungDX/BLOCKS_NORM_MACENKO...No errors so far but I am a bit worried since the whole process is runing since yersterday, I am not able to find any process related to this and
nvidia-smilooks empty. Is this normal? Or it just felt silently? Any idea? Thanks
I would suggest to comment this line imgsList = [i for i in imgsList if not len(os.listdir(i)) == 0] in the data_utils script. It is just used to make sure that the folders containing tiles are not empty and it usually takes time if the number of WSIs are high! maybe you can just ignore this check for first run!
@narminGhaffari, thanks for the feedback. Everything seems to work now. I has to modify most of the paths (from
\\to/), since (I suppose), you were working with Windows. I started a work with ~1300 WSI using resnet with default parametres. Here's theReport.txt:********************************************************************** Namespace(adressExp='/mnt/isilon/Lung_TCGA/train_Level1/ExperimentFile_resnet_default.txt', batch_size=128, clini_dir=['/mnt/isilon/Lung_TCGA/train_Level1/LungDX/LungDX_CLINI.xlsx'], csv_name='CLEANED_DATA', datadir_train=['/mnt/isilon/Lung_TCGA/train_Level1/LungDX/BLOCKS_NORM_MACENKO'], feat_dir=['/mnt/isilon/Lung_TCGA/train_Level1/LungDX/FEATURES'], freeze_Ratio=0.5, gpuNo=0, k=3, lr=0.0001, maxBlockNum=512, max_epochs=20, model_name='resnet', numHighScoreBlocks=20, numHighScorePatients=10, opt='adam', projectFolder='/mnt/isilon/Lung_TCGA/train_Level1/ExperimentFile_resnet_default_lung_type', project_name='ExperimentFile_resnet_default', reg=1e-05, seed=1, slide_dir=['/mnt/isilon/Lung_TCGA/train_Level1/LungDX/LungDX_SLIDE.csv'], target_label='lung_type', target_labels=['lung_type'], train_full=False, useClassicModel=True) ********************************************************************** LOADING DATA FROM/mnt/isilon/Lung_TCGA/train_Level1/LungDX/BLOCKS_NORM_MACENKO...No errors so far but I am a bit worried since the whole process is runing since yersterday, I am not able to find any process related to this and
nvidia-smilooks empty. Is this normal? Or it just felt silently? Any idea? Thanks
@Tato14 Hi Joan. I encountered the same problem as you when using a Linux system to run their code. Could you please give me some hit on how to make the program generate meaningful results? Now it can run without any error output. Thank you so much!