Issues with train.py script
Closed this issue · 10 comments
Hi,
When running the train.py script, it looks like there is an issue with rdkit's multiprocessing of the descriptors:
python train.py --save_dir models/mike/ --compound_csv data/r_al_final_public.csv --rating_col label --val_size 0.2 --n_epochs 10 --featurizer_name morgan_count_rdkit_2d --num_workers 10
/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/utilities/seed.py:55: UserWarning: No seed found, seed set to 2600799618
rank_zero_warn(f"No seed found, seed set to {seed}")
Global seed set to 2600799618
[2023-02-28 10:53:20,895] [INFO] [cleaners.py:111:ensure_readability_and_remove] Checking SMILES validity and removing unreadable ones...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5276/5276 [00:01<00:00, 3641.47it/s]
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Missing logger folder: models/mike/default
| Name | Type | Params
---------------------------------
0 | net | RankNet | 766 K
---------------------------------
766 K Trainable params
0 Non-trainable params
766 K Total params
3.065 Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]Traceback (most recent call last):
File "/Users/michaelcarter/DD_tools/molskill/train.py", line 206, in <module>
train_ranknet(
File "/Users/michaelcarter/DD_tools/molskill/train.py", line 109, in train_ranknet
trainer.fit(
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 768, in fit
self._call_and_handle_interrupt(
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in _fit_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1234, in _run
results = self._run_stage()
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1321, in _run_stage
return self._run_train()
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1343, in _run_train
self._run_sanity_check()
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1411, in _run_sanity_check
val_loop.run()
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 154, in advance
dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 199, in run
self.on_run_start(*args, **kwargs)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 87, in on_run_start
self._data_fetcher = iter(data_fetcher)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 178, in __iter__
self.dataloader_iter = iter(self.dataloader)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 368, in __iter__
return self._get_iterator()
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 314, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 927, in __init__
w.start()
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'Boost.Python.function' object
I think this might be related to this rdkit post rdkit/rdkit#2529
Thanks
I'm also seeing this error with the score.py script:
python score.py --model_ckpt models/default/checkpoints/last.ckpt --featurizer_name rdkit_2d_norm --compound_csv test.csv --smiles_col SMILES --output_csv test_out.csv
[2023-02-28 11:04:47,090] [INFO] [cleaners.py:111:ensure_readability_and_remove] Checking SMILES validity and removing unreadable ones...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 845.37it/s]
Traceback (most recent call last):
File "/Users/michaelcarter/DD_tools/molskill/score.py", line 61, in <module>
LitRankNet.load_from_checkpoint(args.model_ckpt, input_size=featurizer.dim())
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/core/saving.py", line 161, in load_from_checkpoint
model = cls._load_model_state(checkpoint, strict=strict, **kwargs)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/core/saving.py", line 209, in _load_model_state
keys = model.load_state_dict(checkpoint["state_dict"], strict=strict)
File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1497, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for LitRankNet:
size mismatch for net.encoder.0.weight: copying a param with shape torch.Size([256, 2220]) from checkpoint, the shape in current model is torch.Size([256, 172]).
I thought it may have been the --featurizer_name
but it throws the same error for all featurizers.
The output from pytest also confirms my comment above:
FAILED tests/test_scorer.py::test_scorer[1] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_scorer.py::test_scorer[25] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[rdkit_2d] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[rdkit_2d_norm] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[morgan] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[morgan_count] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[avalon] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[avalon_count] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[morgan_count_rdkit_2d] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[morgan_count_rdkit_2d_norm] - TypeError: cannot pickle 'Boost.Python.function' object
Hi @MKCarter
Many thanks for using our software :)
Could you provide a few more details on where you're running our code? Does this happen with the conda package or with the environment file? If so, are you using a fresh environment?
I am not using the conda package as I am running on an M1 Mac, I made the env using:
conda create -n molskills python=3.9
I then pip installed all the requirements from the environment.yml
I think this error is caused by parallelisation. From what I have been told C++ boost functions surfaced as Python import can't be pickled by pickle
, which is used to pass them around the processes. If the code is using multiprocessing
you can try swapping that with multiprocess
, which uses dill
instead of pickle
and can handle Boost functions. The modules are exactly the same, so it's just a matter of swapping the module you are importing.
Cheers
If I do try to conda install this is the error I see BTW:
conda install molskill -c msr-ai4science -c conda-forge
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from current channels:
- molskill
Current channels:
- https://conda.anaconda.org/msr-ai4science/osx-64
- https://conda.anaconda.org/msr-ai4science/noarch
- https://conda.anaconda.org/conda-forge/osx-64
- https://conda.anaconda.org/conda-forge/noarch
- https://conda.anaconda.org/omnia/osx-64
- https://conda.anaconda.org/omnia/noarch
- https://repo.anaconda.com/pkgs/main/osx-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/r/osx-64
- https://repo.anaconda.com/pkgs/r/noarch
Hi @MKCarter. We currently only support Linux 3.9 builds (we state this in the README.md
file). I cannot reproduce your issue on a clean env on a Linux machine:
(base) REDMOND.jjimenezluna@XXX:~/molskill$ mamba env create -n molskill-test -f environment.yml python=3.9
pytorch/linux-64 [====================] (00m:00s) No change
pytorch/noarch [====================] (00m:00s) Done
pkgs/r/noarch [====================] (00m:00s) No change
pkgs/r/linux-64 [====================] (00m:00s) No change
pkgs/main/noarch [====================] (00m:00s) Done
pkgs/main/linux-64 [====================] (00m:00s) Done
conda-forge/noarch [====================] (00m:02s) Done
conda-forge/linux-64 [====================] (00m:05s) Done
Looking for: ['python=3.9', 'pip=22.1.2', 'numpy=1.23.1', 'pandas=1.4.2', 'scipy=1.8.1', 'rdkit=2022.03.3', 'tqdm=4.64.0', 'scikit-learn=1.0.1', 'pytorch=1.11', 'cudatoolkit=11.3', 'black=22.6.0', 'pre-commit=2.20.0', 'pyright=1.1.269']
Transaction
Prefix: /home/jjimenezluna/miniconda3/envs/molskill-test
Updating specs:
- python=3.9
- pip=22.1.2
- numpy=1.23.1
- pandas=1.4.2
- scipy=1.8.1
- rdkit=2022.03.3
- tqdm=4.64.0
- scikit-learn=1.0.1
- pytorch=1.11
- cudatoolkit=11.3
- black=22.6.0
- pre-commit=2.20.0
- pyright=1.1.269
Package Version Build Channel Size
────────────────────────────────────────────────────────────────────────────────────────────────────
Install:
────────────────────────────────────────────────────────────────────────────────────────────────────
+ _libgcc_mutex 0.1 conda_forge conda-forge/linux-64 Cached
+ _openmp_mutex 4.5 2_gnu conda-forge/linux-64 Cached
+ black 22.6.0 py39hf3d152e_2 conda-forge/linux-64 Cached
+ boost 1.74.0 py39h5472131_5 conda-forge/linux-64 Cached
+ boost-cpp 1.74.0 h75c5d50_8 conda-forge/linux-64 Cached
+ brotli 1.0.9 h166bdaf_8 conda-forge/linux-64 Cached
+ brotli-bin 1.0.9 h166bdaf_8 conda-forge/linux-64 Cached
+ bzip2 1.0.8 h7f98852_4 conda-forge/linux-64 Cached
+ ca-certificates 2022.12.7 ha878542_0 conda-forge/linux-64 Cached
+ cairo 1.16.0 ha61ee94_1014 conda-forge/linux-64 Cached
+ certifi 2022.12.7 pyhd8ed1ab_0 conda-forge/noarch Cached
+ cffi 1.15.1 py39he91dace_3 conda-forge/linux-64 Cached
+ cfgv 3.3.1 pyhd8ed1ab_0 conda-forge/noarch Cached
+ click 8.1.3 py39hf3d152e_1 conda-forge/linux-64 Cached
+ colorama 0.4.6 pyhd8ed1ab_0 conda-forge/noarch Cached
+ contourpy 1.0.7 py39h4b4f3f3_0 conda-forge/linux-64 210 KB
+ cudatoolkit 11.3.1 h9edb442_11 conda-forge/linux-64 Cached
+ cudnn 8.4.1.50 hed8a83a_0 conda-forge/linux-64 Cached
+ cycler 0.11.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ distlib 0.3.6 pyhd8ed1ab_0 conda-forge/noarch Cached
+ expat 2.5.0 h27087fc_0 conda-forge/linux-64 Cached
+ filelock 3.9.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge/noarch Cached
+ font-ttf-inconsolata 3.000 h77eed37_0 conda-forge/noarch Cached
+ font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge/noarch Cached
+ font-ttf-ubuntu 0.83 hab24e00_0 conda-forge/noarch Cached
+ fontconfig 2.14.2 h14ed4e7_0 conda-forge/linux-64 Cached
+ fonts-conda-ecosystem 1 0 conda-forge/noarch Cached
+ fonts-conda-forge 1 0 conda-forge/noarch Cached
+ fonttools 4.38.0 py39hb9d737c_1 conda-forge/linux-64 Cached
+ freetype 2.12.1 hca18f0e_1 conda-forge/linux-64 Cached
+ gettext 0.21.1 h27087fc_0 conda-forge/linux-64 Cached
+ greenlet 2.0.2 py39h227be39_0 conda-forge/linux-64 186 KB
+ icu 70.1 h27087fc_0 conda-forge/linux-64 Cached
+ identify 2.5.18 pyhd8ed1ab_0 conda-forge/noarch 76 KB
+ importlib-metadata 6.0.0 pyha770c72_0 conda-forge/noarch Cached
+ importlib-resources 5.12.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ importlib_resources 5.12.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ intel-openmp 2022.1.0 h9e868ea_3769 pkgs/main/linux-64 Cached
+ joblib 1.2.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ jpeg 9e h0b41bf4_3 conda-forge/linux-64 Cached
+ kiwisolver 1.4.4 py39hf939315_1 conda-forge/linux-64 Cached
+ lcms2 2.14 hfd0df8a_1 conda-forge/linux-64 Cached
+ ld_impl_linux-64 2.40 h41732ed_0 conda-forge/linux-64 Cached
+ lerc 4.0.0 h27087fc_0 conda-forge/linux-64 Cached
+ libblas 3.9.0 16_linux64_mkl conda-forge/linux-64 Cached
+ libbrotlicommon 1.0.9 h166bdaf_8 conda-forge/linux-64 Cached
+ libbrotlidec 1.0.9 h166bdaf_8 conda-forge/linux-64 Cached
+ libbrotlienc 1.0.9 h166bdaf_8 conda-forge/linux-64 Cached
+ libcblas 3.9.0 16_linux64_mkl conda-forge/linux-64 Cached
+ libdeflate 1.17 h0b41bf4_0 conda-forge/linux-64 Cached
+ libffi 3.4.2 h7f98852_5 conda-forge/linux-64 Cached
+ libgcc-ng 12.2.0 h65d4601_19 conda-forge/linux-64 Cached
+ libgfortran-ng 12.2.0 h69a702a_19 conda-forge/linux-64 Cached
+ libgfortran5 12.2.0 h337968e_19 conda-forge/linux-64 Cached
+ libglib 2.74.1 h606061b_1 conda-forge/linux-64 Cached
+ libgomp 12.2.0 h65d4601_19 conda-forge/linux-64 Cached
+ libiconv 1.17 h166bdaf_0 conda-forge/linux-64 Cached
+ liblapack 3.9.0 16_linux64_mkl conda-forge/linux-64 Cached
+ libnsl 2.0.0 h7f98852_0 conda-forge/linux-64 Cached
+ libpng 1.6.39 h753d276_0 conda-forge/linux-64 Cached
+ libprotobuf 3.20.2 h6239696_0 conda-forge/linux-64 Cached
+ libsqlite 3.40.0 h753d276_0 conda-forge/linux-64 Cached
+ libstdcxx-ng 12.2.0 h46fd767_19 conda-forge/linux-64 Cached
+ libtiff 4.5.0 h6adf6a1_2 conda-forge/linux-64 Cached
+ libuuid 2.32.1 h7f98852_1000 conda-forge/linux-64 Cached
+ libuv 1.44.2 h166bdaf_0 conda-forge/linux-64 Cached
+ libwebp-base 1.2.4 h166bdaf_0 conda-forge/linux-64 Cached
+ libxcb 1.13 h7f98852_1004 conda-forge/linux-64 Cached
+ libzlib 1.2.13 h166bdaf_4 conda-forge/linux-64 Cached
+ magma 2.5.4 hc72dce7_4 conda-forge/linux-64 Cached
+ matplotlib-base 3.7.0 py39he190548_0 conda-forge/linux-64 6 MB
+ mkl 2022.1.0 hc2b9512_224 pkgs/main/linux-64 Cached
+ munkres 1.1.4 pyh9f0ad1d_0 conda-forge/noarch Cached
+ mypy_extensions 1.0.0 pyha770c72_0 conda-forge/noarch Cached
+ nccl 2.14.3.1 h0800d71_0 conda-forge/linux-64 Cached
+ ncurses 6.3 h27087fc_1 conda-forge/linux-64 Cached
+ ninja 1.11.1 h924138e_0 conda-forge/linux-64 2 MB
+ nodeenv 1.7.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ nodejs 18.13.0 h8d033a5_0 conda-forge/linux-64 14 MB
+ numpy 1.23.1 py39hba7629e_0 conda-forge/linux-64 Cached
+ openjpeg 2.5.0 hfec8fc6_2 conda-forge/linux-64 Cached
+ openssl 3.0.8 h0b41bf4_0 conda-forge/linux-64 Cached
+ packaging 23.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ pandas 1.4.2 py39h1832856_2 conda-forge/linux-64 Cached
+ pathspec 0.11.0 pyhd8ed1ab_0 conda-forge/noarch 36 KB
+ pcre2 10.40 hc3806b6_0 conda-forge/linux-64 Cached
+ pillow 9.4.0 py39h2320bf1_1 conda-forge/linux-64 44 MB
+ pip 22.1.2 pyhd8ed1ab_0 conda-forge/noarch Cached
+ pixman 0.40.0 h36c2ea0_0 conda-forge/linux-64 Cached
+ platformdirs 3.0.0 pyhd8ed1ab_0 conda-forge/noarch 17 KB
+ pre-commit 2.20.0 py39hf3d152e_1 conda-forge/linux-64 Cached
+ pthread-stubs 0.4 h36c2ea0_1001 conda-forge/linux-64 Cached
+ pycairo 1.23.0 py39h23c5bb2_0 conda-forge/linux-64 Cached
+ pycparser 2.21 pyhd8ed1ab_0 conda-forge/noarch Cached
+ pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge/noarch Cached
+ pyright 1.1.269 py39hb9d737c_0 conda-forge/linux-64 Cached
+ python 3.9.16 h2782a2a_0_cpython conda-forge/linux-64 23 MB
+ python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge/noarch Cached
+ python_abi 3.9 3_cp39 conda-forge/linux-64 Cached
+ pytorch 1.11.0 cuda112py39ha0cca9b_202 conda-forge/linux-64 Cached
+ pytz 2022.7.1 pyhd8ed1ab_0 conda-forge/noarch Cached
+ pyyaml 6.0 py39hb9d737c_5 conda-forge/linux-64 Cached
+ rdkit 2022.03.3 py39h89e00b9_0 conda-forge/linux-64 Cached
+ readline 8.1.2 h0f457ee_0 conda-forge/linux-64 Cached
+ reportlab 3.6.12 py39h8bbe095_2 conda-forge/linux-64 Cached
+ scikit-learn 1.0.1 py39h4dfa638_3 conda-forge/linux-64 Cached
+ scipy 1.8.1 py39hddc5342_3 conda-forge/linux-64 Cached
+ setuptools 67.4.0 pyhd8ed1ab_0 conda-forge/noarch 567 KB
+ six 1.16.0 pyh6c4a22f_0 conda-forge/noarch Cached
+ sleef 3.5.1 h9b69904_2 conda-forge/linux-64 Cached
+ sqlalchemy 2.0.4 py39h72bdee0_0 conda-forge/linux-64 2 MB
+ threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge/noarch Cached
+ tk 8.6.12 h27826a3_0 conda-forge/linux-64 Cached
+ toml 0.10.2 pyhd8ed1ab_0 conda-forge/noarch Cached
+ tomli 2.0.1 pyhd8ed1ab_0 conda-forge/noarch Cached
+ tqdm 4.64.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ typing-extensions 4.4.0 hd8ed1ab_0 conda-forge/noarch Cached
+ typing_extensions 4.4.0 pyha770c72_0 conda-forge/noarch Cached
+ tzdata 2022g h191b570_0 conda-forge/noarch Cached
+ ukkonen 1.0.1 py39hf939315_3 conda-forge/linux-64 Cached
+ unicodedata2 15.0.0 py39hb9d737c_0 conda-forge/linux-64 Cached
+ virtualenv 20.19.0 pyhd8ed1ab_0 conda-forge/noarch 6 MB
+ wheel 0.38.4 pyhd8ed1ab_0 conda-forge/noarch Cached
+ xorg-kbproto 1.0.7 h7f98852_1002 conda-forge/linux-64 Cached
+ xorg-libice 1.0.10 h7f98852_0 conda-forge/linux-64 Cached
+ xorg-libsm 1.2.3 hd9c2040_1000 conda-forge/linux-64 Cached
+ xorg-libx11 1.7.2 h7f98852_0 conda-forge/linux-64 Cached
+ xorg-libxau 1.0.9 h7f98852_0 conda-forge/linux-64 Cached
+ xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge/linux-64 Cached
+ xorg-libxext 1.3.4 h0b41bf4_2 conda-forge/linux-64 Cached
+ xorg-libxrender 0.9.10 h7f98852_1003 conda-forge/linux-64 Cached
+ xorg-renderproto 0.11.1 h7f98852_1002 conda-forge/linux-64 Cached
+ xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge/linux-64 Cached
+ xorg-xproto 7.0.31 h7f98852_1007 conda-forge/linux-64 Cached
+ xz 5.2.6 h166bdaf_0 conda-forge/linux-64 Cached
+ yaml 0.2.5 h7f98852_2 conda-forge/linux-64 Cached
+ zipp 3.15.0 pyhd8ed1ab_0 conda-forge/noarch 17 KB
+ zlib 1.2.13 h166bdaf_4 conda-forge/linux-64 Cached
+ zstd 1.5.2 h3eb15da_6 conda-forge/linux-64 Cached
Summary:
Install: 140 packages
Total download: 100 MB
────────────────────────────────────────────────────────────────────────────────────────────────────
Downloading and Extracting Packages
setuptools-67.4.0 | 567 KB | ####################################################### | 100%
ninja-1.11.1 | 2.1 MB | ####################################################### | 100%
python-3.9.16 | 23.0 MB | ####################################################### | 100%
pathspec-0.11.0 | 36 KB | ####################################################### | 100%
platformdirs-3.0.0 | 17 KB | ####################################################### | 100%
contourpy-1.0.7 | 210 KB | ####################################################### | 100%
nodejs-18.13.0 | 14.4 MB | ####################################################### | 100%
pillow-9.4.0 | 44.1 MB | ####################################################### | 100%
identify-2.5.18 | 76 KB | ####################################################### | 100%
virtualenv-20.19.0 | 6.0 MB | ####################################################### | 100%
greenlet-2.0.2 | 186 KB | ####################################################### | 100%
matplotlib-base-3.7. | 6.4 MB | ####################################################### | 100%
zipp-3.15.0 | 17 KB | ####################################################### | 100%
sqlalchemy-2.0.4 | 2.4 MB | ####################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: / By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the CUDA End User License Agreement (EULA): https://docs.nvidia.com/cuda/eula/index.html
\ By downloading and using the cuDNN conda packages, you accept the terms and conditions of the NVIDIA cuDNN EULA -
https://docs.nvidia.com/deeplearning/cudnn/sla/index.html
done
Installing pip dependencies: | Ran pip subprocess with arguments:
['/home/jjimenezluna/miniconda3/envs/molskill-test/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt']
Pip subprocess output:
Collecting pytorch-lightning==1.6.3
Using cached pytorch_lightning-1.6.3-py3-none-any.whl (584 kB)
Collecting torchmetrics>=0.4.1
Downloading torchmetrics-0.11.1-py3-none-any.whl (517 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 517.2/517.2 kB 2.2 MB/s eta 0:00:00
Requirement already satisfied: tqdm>=4.57.0 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (4.64.0)
Collecting pyDeprecate<0.4.0,>=0.3.1
Using cached pyDeprecate-0.3.2-py3-none-any.whl (10 kB)
Requirement already satisfied: packaging>=17.0 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (23.0)
Requirement already satisfied: torch>=1.8.* in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (1.11.0.post202)
Requirement already satisfied: PyYAML>=5.4 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (6.0)
Requirement already satisfied: typing-extensions>=4.0.0 in /home/jjimenezluna/.local/lib/python3.9/site-packages (from pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (4.3.0)
Collecting fsspec[http]!=2021.06.0,>=2021.05.0
Downloading fsspec-2023.1.0-py3-none-any.whl (143 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.0/143.0 kB 18.6 MB/s eta 0:00:00
Collecting tensorboard>=2.2.0
Downloading tensorboard-2.12.0-py3-none-any.whl (5.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.6/5.6 MB 25.9 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.17.2 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (1.23.1)
Collecting aiohttp!=4.0.0a0,!=4.0.0a1
Downloading aiohttp-3.8.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 59.9 MB/s eta 0:00:00
Collecting requests
Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting protobuf>=3.19.6
Downloading protobuf-4.22.0-cp37-abi3-manylinux2014_x86_64.whl (302 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 302.4/302.4 kB 34.5 MB/s eta 0:00:00
Collecting markdown>=2.6.8
Using cached Markdown-3.4.1-py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.6.3
Downloading google_auth-2.16.1-py2.py3-none-any.whl (177 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 177.2/177.2 kB 26.8 MB/s eta 0:00:00
Collecting grpcio>=1.48.2
Downloading grpcio-1.51.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.8/4.8 MB 43.7 MB/s eta 0:00:00
Collecting tensorboard-data-server<0.8.0,>=0.7.0
Downloading tensorboard_data_server-0.7.0-py3-none-manylinux2014_x86_64.whl (6.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.6/6.6 MB 43.5 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=41.0.0 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (67.4.0)
Collecting werkzeug>=1.0.1
Downloading Werkzeug-2.2.3-py3-none-any.whl (233 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 233.6/233.6 kB 32.4 MB/s eta 0:00:00
Requirement already satisfied: wheel>=0.26 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (0.38.4)
Collecting absl-py>=0.4
Using cached absl_py-1.4.0-py3-none-any.whl (126 kB)
Collecting tensorboard-plugin-wit>=1.6.0
Using cached tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
Collecting google-auth-oauthlib<0.5,>=0.4.1
Using cached google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Collecting attrs>=17.3.0
Using cached attrs-22.2.0-py3-none-any.whl (60 kB)
Collecting async-timeout<5.0,>=4.0.0a3
Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting charset-normalizer<4.0,>=2.0
Using cached charset_normalizer-3.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (198 kB)
Collecting frozenlist>=1.1.1
Using cached frozenlist-1.3.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (158 kB)
Collecting aiosignal>=1.1.2
Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting multidict<7.0,>=4.5
Using cached multidict-6.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)
Collecting yarl<2.0,>=1.0
Using cached yarl-1.8.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (264 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.9-py3-none-any.whl (34 kB)
Requirement already satisfied: six>=1.9.0 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (1.16.0)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting cachetools<6.0,>=2.0.0
Downloading cachetools-5.3.0-py3-none-any.whl (9.3 kB)
Collecting requests-oauthlib>=0.7.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Requirement already satisfied: importlib-metadata>=4.4 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from markdown>=2.6.8->tensorboard>=2.2.0->pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (6.0.0)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.14-py2.py3-none-any.whl (140 kB)
Collecting idna<4,>=2.5
Using cached idna-3.4-py3-none-any.whl (61 kB)
Requirement already satisfied: certifi>=2017.4.17 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from requests->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (2022.12.7)
Collecting MarkupSafe>=2.1.1
Downloading MarkupSafe-2.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Requirement already satisfied: zipp>=0.5 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard>=2.2.0->pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (3.15.0)
Collecting pyasn1<0.5.0,>=0.4.6
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.2-py3-none-any.whl (151 kB)
Installing collected packages: tensorboard-plugin-wit, pyasn1, charset-normalizer, urllib3, tensorboard-data-server, rsa, pyDeprecate, pyasn1-modules, protobuf, oauthlib, multidict, MarkupSafe, idna, grpcio, fsspec, frozenlist, cachetools, attrs, async-timeout, absl-py, yarl, werkzeug, torchmetrics, requests, markdown, google-auth, aiosignal, requests-oauthlib, aiohttp, google-auth-oauthlib, tensorboard, pytorch-lightning
Successfully installed MarkupSafe-2.1.2 absl-py-1.4.0 aiohttp-3.8.4 aiosignal-1.3.1 async-timeout-4.0.2 attrs-22.2.0 cachetools-5.3.0 charset-normalizer-3.0.1 frozenlist-1.3.3 fsspec-2023.1.0 google-auth-2.16.1 google-auth-oauthlib-0.4.6 grpcio-1.51.3 idna-3.4 markdown-3.4.1 multidict-6.0.4 oauthlib-3.2.2 protobuf-4.22.0 pyDeprecate-0.3.2 pyasn1-0.4.8 pyasn1-modules-0.2.8 pytorch-lightning-1.6.3 requests-2.28.2 requests-oauthlib-1.3.1 rsa-4.9 tensorboard-2.12.0 tensorboard-data-server-0.7.0 tensorboard-plugin-wit-1.8.1 torchmetrics-0.11.1 urllib3-1.26.14 werkzeug-2.2.3 yarl-1.8.2
done
#
# To activate this environment, use
#
# $ conda activate molskill-test
#
# To deactivate an active environment, use
#
# $ conda deactivate
(base) REDMOND.jjimenezluna@XXX:~/molskill$ conda activate molskill-test
(molskill-test) REDMOND.jjimenezluna@XXX:~/molskill$ pip install pytest
Collecting pytest
Downloading pytest-7.2.1-py3-none-any.whl (317 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 317.1/317.1 kB 1.6 MB/s eta 0:00:00
Collecting iniconfig
Using cached iniconfig-2.0.0-py3-none-any.whl (5.9 kB)
Requirement already satisfied: attrs>=19.2.0 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from pytest) (22.2.0)
Collecting exceptiongroup>=1.0.0rc8
Using cached exceptiongroup-1.1.0-py3-none-any.whl (14 kB)
Requirement already satisfied: tomli>=1.0.0 in /home/jjimenezluna/.local/lib/python3.9/site-packages (from pytest) (2.0.1)
Collecting pluggy<2.0,>=0.12
Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Requirement already satisfied: packaging in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from pytest) (23.0)
Installing collected packages: pluggy, iniconfig, exceptiongroup, pytest
Successfully installed exceptiongroup-1.1.0 iniconfig-2.0.0 pluggy-1.0.0 pytest-7.2.1
(molskill-test) REDMOND.jjimenezluna@XXX:~/molskill$ export PYTHONPATH=.
(molskill-test) REDMOND.jjimenezluna@XXX:~/molskill$ pytest
====================================== test session starts =======================================
platform linux -- Python 3.9.16, pytest-7.2.1, pluggy-1.0.0
rootdir: /home/jjimenezluna/molskill
collected 21 items
tests/test_featurizers.py .......... [ 47%]
tests/test_links.py . [ 52%]
tests/test_scorer.py .. [ 61%]
tests/test_train.py ........ [100%]
======================================== warnings summary ========================================
../miniconda3/envs/molskill-test/lib/python3.9/site-packages/pkg_resources/__init__.py:2804
/home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/pkg_resources/__init__.py:2804: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
../miniconda3/envs/molskill-test/lib/python3.9/site-packages/pkg_resources/__init__.py:2804
/home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/pkg_resources/__init__.py:2804: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
../miniconda3/envs/molskill-test/lib/python3.9/site-packages/torch/utils/tensorboard/__init__.py:4
../miniconda3/envs/molskill-test/lib/python3.9/site-packages/torch/utils/tensorboard/__init__.py:4
/home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/torch/utils/tensorboard/__init__.py:4: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if not hasattr(tensorboard, '__version__') or LooseVersion(tensorboard.__version__) < LooseVersion('1.15'):
../miniconda3/envs/molskill-test/lib/python3.9/site-packages/pytorch_lightning/__init__.py:38
/home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/pytorch_lightning/__init__.py:38: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pytorch_lightning')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
__import__("pkg_resources").declare_namespace(__name__)
../miniconda3/envs/molskill-test/lib/python3.9/site-packages/sklearn/utils/multiclass.py:14
/home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/sklearn/utils/multiclass.py:14: DeprecationWarning: Please use `spmatrix` from the `scipy.sparse` namespace, the `scipy.sparse.base` namespace is deprecated.
from scipy.sparse.base import spmatrix
tests/test_scorer.py::test_scorer[1]
/home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/torch/distributed/_sharded_tensor/__init__.py:8: DeprecationWarning: torch.distributed._sharded_tensor will be deprecated, use torch.distributed._shard.sharded_tensor instead
warnings.warn(
tests/test_train.py::test_loss_goes_down[rdkit_2d]
tests/test_train.py::test_loss_goes_down[rdkit_2d_norm]
tests/test_train.py::test_loss_goes_down[morgan]
tests/test_train.py::test_loss_goes_down[morgan_count]
tests/test_train.py::test_loss_goes_down[avalon]
tests/test_train.py::test_loss_goes_down[avalon_count]
tests/test_train.py::test_loss_goes_down[morgan_count_rdkit_2d]
tests/test_train.py::test_loss_goes_down[morgan_count_rdkit_2d_norm]
/home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/pytorch_lightning/trainer/configuration_validator.py:133: UserWarning: You defined a `validation_step` but have no `val_dataloader`. Skipping val loop.
rank_zero_warn("You defined a `validation_step` but have no `val_dataloader`. Skipping val loop.")
tests/test_train.py::test_loss_goes_down[rdkit_2d]
tests/test_train.py::test_loss_goes_down[rdkit_2d_norm]
tests/test_train.py::test_loss_goes_down[morgan]
tests/test_train.py::test_loss_goes_down[morgan_count]
tests/test_train.py::test_loss_goes_down[avalon]
tests/test_train.py::test_loss_goes_down[avalon_count]
tests/test_train.py::test_loss_goes_down[morgan_count_rdkit_2d]
tests/test_train.py::test_loss_goes_down[morgan_count_rdkit_2d_norm]
/home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:1927: PossibleUserWarning: The number of training batches (2) is smaller than the logging interval Trainer(log_every_n_steps=20). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
rank_zero_warn(
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ 21 passed, 23 warnings in 40.79s ================================
I believe your issue lies on how the multiprocessing is handled by the torch DataLoader
across different platforms.
Can you manually change the num_workers=1
arg in the MolSkillScorer
instance and see whether it works?
@josejimenezluna I tried, but no luck! Thanks for the suggestion and the help.
I will take a look and see if there is anyway to make this Mac compatible.
If I find a solution, I will update you.
Cheers
Thinking about it, the easiest solution is to use docker...
Hi @MKCarter. I am closing this since we do not currently plan on providing OS X support.
An easy (inefficient) fix for this is to set scorer = MolSkillScorer(num_workers=0)
, which disables multiprocessing.