Question about path_mean and path_std
Closed this issue · 6 comments
Hello! Thanks for your code! But I've been meeting a confusing problem about path_mean and path_std. In the training process, I noticed the values of path_mean and path_std were confusing, but the model could still be trained but very slowly. What's wrong with it? I'd appreciate it if you could help me! Thank you!
path_mean: [-4.436188726613424e-32, 3.0744488307286487e-41, -4.436184024636021e-32] path_std: [nan, 5.54477138001159e-21, nan]
############### CONFIGURATION ############### all_wsi: all_wsi val_wsi_per_class: 20 test_wsi_per_class: 30 keep_orig_copy: True num_workers: 8 patch_size: 224 wsi_train: wsi_train wsi_val: wsi_val wsi_test: wsi_test labels_train: labels_train.csv labels_val: labels_val.csv labels_test: labels_test.csv train_folder: train_folder patches_eval_train: patches_eval_train patches_eval_val: patches_eval_val patches_eval_test: patches_eval_test num_train_per_class: 80000 type_histopath: True purple_threshold: 100 purple_scale_size: 15 slide_overlap: 3 gen_val_patches_overlap_factor: 1.5 image_ext: jpg by_folder: True color_jitter_brightness: 0.5 color_jitter_contrast: 0.5 color_jitter_saturation: 0.5 color_jitter_hue: 0.2 num_epochs: 100 num_layers: 18 learning_rate: 0.001 batch_size: 128 weight_decay: 0.0001 learning_rate_decay: 0.85 resume_checkpoint: False save_interval: 1 checkpoints_folder: checkpoints checkpoint_file: xyz.pt pretrain: False log_folder: logs auto_select: True preds_train: preds_train preds_val: preds_val preds_test: preds_test inference_train: inference_train inference_val: inference_val inference_test: inference_test vis_train: vis_train vis_val: vis_val vis_test: vis_test device: cuda:0 classes: ['N', 'Y'] num_classes: 2 train_patches: train_folder/train val_patches: train_folder/val path_mean: [-4.436188726613424e-32, 3.0744488307286487e-41, -4.436184024636021e-32] path_std: [nan, 5.54477138001159e-21, nan] resume_checkpoint_path: checkpoints/xyz.pt log_csv: logs/log_12182020_123013.csv eval_model: checkpoints/xyz.pt threshold_search: (0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9) colors: ('red', 'white', 'blue', 'green', 'purple', 'orange', 'black', 'pink', 'yellow')
The path_mean and path_std are definitely a problem. Based on the numbers you provided, I think the directory all_wsi is empty. Can you tell me how you have your files stored in the code directory?
The path_mean and path_std are definitely a problem. Based on the numbers you provided, I think the directory
all_wsiis empty. Can you tell me how you have your files stored in the code directory?
Thanks for your reply! I checked the format of my images, they are 'PNG'! I made some adjustments and now it looks good! Thanks for your help!
But when I ran the '3_train.py', another problem of the confusing matrix came out. I don't know what goes wrong.
AttributeError: 'Styler' object has no attribute 'hide_index'
Traceback (most recent call last): File "3_train.py", line 28, in <module> weight_decay=config.args.weight_decay) File "/media/move_eric/My_Passport/pathology_drug_resistence_recognition/MILtry/deepslidetry/code/utils_model.py", line 498, in train_resnet num_classes=num_classes) File "/media/move_eric/My_Passport/pathology_drug_resistence_recognition/MILtry/deepslidetry/code/utils_model.py", line 285, in train_helper num_classes=num_classes) File "/media/move_eric/My_Passport/pathology_drug_resistence_recognition/MILtry/deepslidetry/code/utils_model.py", line 63, in calculate_confusion_matrix cm.style.hide_index() AttributeError: 'Styler' object has no attribute 'hide_index'
Can you provide a list of the installed Python packages and their versions? This appears to be a problem with the version you have installed.
Can you provide a list of the installed Python packages and their versions? This appears to be a problem with the version you have installed.
Sure!
# Name Version _libgcc_mutex 0.1 absl-py 0.11.0 aiohttp 3.7.3 async-timeout 3.0.1 attrs 20.3.0 blas 1.0 blinker 1.4 brotlipy 0.7.0 c-ares 1.17.1 ca-certificates 2020.12.8 cachetools 4.2.0 certifi 2020.12.5 cffi 1.14.4 chardet 3.0.4 click 7.1.2 cryptography 3.3.1 cudatoolkit 10.1.243 cycler 0.10.0 dbus 1.13.18 decorator 4.4.2 docopt 0.6.2 expat 2.2.10 fontconfig 2.13.0 freetype 2.10.4 future 0.18.2 glib 2.66.1 google-auth 1.24.0 google-auth-oauthlib 0.4.2 grpcio 1.31.0 gst-plugins-base 1.14.0 gstreamer 1.14.0 icu 58.2 idna 2.10 idna_ssl 1.1.0 imageio 2.9.0 imgaug 0.4.0 importlib-metadata 2.0.0 intel-openmp 2020.2 jinja2 2.11.2 jpeg 9b jsonpickle 0.9.6 kiwisolver 1.3.0 lcms2 2.11 ld_impl_linux-64 2.33.1 libedit 3.1.20191231 libffi 3.3 libgcc-ng 9.1.0 libpng 1.6.37 libprotobuf 3.13.0.1 libstdcxx-ng 9.1.0 libtiff 4.1.0 libuuid 1.0.3 libxcb 1.14 libxml2 2.9.10 lz4-c 1.9.2 markdown 3.3.3 markupsafe 1.1.1 matplotlib 3.2.2 matplotlib-base 3.3.2 mkl 2020.2 mkl-service 2.3.0 mkl_fft 1.2.0 mkl_random 1.1.1 multidict 4.7.6 munch 2.5.0 ncurses 6.2 networkx 2.5 ninja 1.10.2 numpy 1.19.2 numpy-base 1.19.2 oauthlib 3.1.0 olefile 0.46 opencv-python 4.4.0.46 openslide-python 1.1.2 openssl 1.1.1i pandas 0.22.0 pcre 8.44 pillow 5.2.0 pip 20.3.1 pretrainedmodels 0.6.2 protobuf 3.14.0 py-cpuinfo 7.0.0 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycparser 2.20 pyjwt 1.7.1 pyopenssl 20.0.1 pyparsing 2.4.7 pyqt 5.9.2 pysocks 1.7.1 python 3.6.12 python-dateutil 2.8.1 python-telegram-bot 10.1.0 pytorch 1.5.1 pytz 2020.4 pywavelets 1.1.1 qt 5.9.7 readline 8.0 requests 2.25.1 requests-oauthlib 1.3.0 rsa 4.6 sacred 0.7.4 scikit-image 0.17.2 scikit-learn 0.19.2 scipy 1.1.0 setuptools 51.0.0 shapely 1.7.1 sip 4.19.8 six 1.15.0 sqlite 3.33.0 tensorboard 2.3.0 tensorboard-plugin-wit 1.6.0 tensorboardx 2.1 tifffile 2020.9.3 tk 8.6.10 torchvision 0.6.1 tornado 6.1 tqdm 4.23.3 typing-extensions 3.7.4.3 typing_extensions 3.7.4.3 urllib3 1.26.2 werkzeug 1.0.1 wheel 0.36.1 wrapt 1.12.1 xz 5.2.5 yarl 1.5.1 zipp 3.4.0 zlib 1.2.11 zstd 1.4.5
The issue is with your pandas version. The hide_index() method requires version 0.23.0 or newer. Alternatively, you could comment out the hide_index() line if you don't want to upgrade.
The issue is with your pandas version. The hide_index() method requires version 0.23.0 or newer. Alternatively, you could comment out the hide_index() line if you don't want to upgrade.
It now works well! Thank you! I really appreciate your help!