Error when loading weights during training
Closed this issue · 1 comments
panyuxin1993 commented
Thank you for this powerful tools. I like it very much and want to train the model using our own data. Before that, I downloaded a DAVIS dataset and want to test model training.
(cutie) tervolab@tervolab-ws1:~/Cutie$ HYDRA_FULL_ERROR=1 OMP_NUM_THREADS=2 torchrun --master_port 25357 --nproc_per_node=2 cutie/train.py exp_id=00 model=base data=base weights=weights/cutie-base-mega.pth
However I get following errors. Could you please give me some hints how to solve the problem?
/home/tervolab/Cutie/cutie/model/trainer.py:229: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
src_dict = torch.load(path, map_location={'cuda:0': map_location})
[2024-08-22 21:29:10][INFO][r1] - Importing network weights from weights/cutie-base-mega.pth...
Error executing job with overrides: ['exp_id=00', 'model=base', 'data=base', 'weights=weights/cutie-base-mega.pth']
[rank1]: Traceback (most recent call last):
[rank1]: File "/home/tervolab/Cutie/cutie/train.py", line 170, in <module>
[rank1]: train()
[rank1]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
[rank1]: _run_hydra(
[rank1]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
[rank1]: _run_app(
[rank1]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
[rank1]: run_and_report(
[rank1]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
[rank1]: raise ex
[rank1]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
[rank1]: return func()
[rank1]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
[rank1]: lambda: hydra.run(
[rank1]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
[rank1]: _ = ret.return_value
[rank1]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
[rank1]: raise self._return_value
[rank1]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
[rank1]: ret.return_value = task_function(task_cfg)
[rank1]: File "/home/tervolab/Cutie/cutie/train.py", line 98, in train
[rank1]: trainer.load_weights(cfg['weights'])
[rank1]: File "/home/tervolab/Cutie/cutie/model/trainer.py", line 232, in load_weights
[rank1]: self.load_weights_in_memory(src_dict)
[rank1]: File "/home/tervolab/Cutie/cutie/model/trainer.py", line 223, in load_weights_in_memory
[rank1]: self.cutie.module.load_weights(src_dict)
[rank1]: File "/home/tervolab/Cutie/cutie/model/cutie.py", line 244, in load_weights
[rank1]: log.warning(f'Converting {k} from multiple objects to single object.'
[rank1]: UnboundLocalError: local variable 'k' referenced before assignment
[2024-08-22 21:29:10][INFO][r0] - Importing network weights from weights/cutie-base-mega.pth...
Error executing job with overrides: ['exp_id=00', 'model=base', 'data=base', 'weights=weights/cutie-base-mega.pth']
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/tervolab/Cutie/cutie/train.py", line 170, in <module>
[rank0]: train()
[rank0]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
[rank0]: _run_hydra(
[rank0]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
[rank0]: _run_app(
[rank0]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
[rank0]: run_and_report(
[rank0]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
[rank0]: raise ex
[rank0]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
[rank0]: return func()
[rank0]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
[rank0]: lambda: hydra.run(
[rank0]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
[rank0]: _ = ret.return_value
[rank0]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
[rank0]: raise self._return_value
[rank0]: File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
[rank0]: ret.return_value = task_function(task_cfg)
[rank0]: File "/home/tervolab/Cutie/cutie/train.py", line 98, in train
[rank0]: trainer.load_weights(cfg['weights'])
[rank0]: File "/home/tervolab/Cutie/cutie/model/trainer.py", line 232, in load_weights
[rank0]: self.load_weights_in_memory(src_dict)
[rank0]: File "/home/tervolab/Cutie/cutie/model/trainer.py", line 223, in load_weights_in_memory
[rank0]: self.cutie.module.load_weights(src_dict)
[rank0]: File "/home/tervolab/Cutie/cutie/model/cutie.py", line 244, in load_weights
[rank0]: log.warning(f'Converting {k} from multiple objects to single object.'
[rank0]: UnboundLocalError: local variable 'k' referenced before assignment
E0822 21:29:10.483000 135417656243264 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 115341) of binary: /home/tervolab/anaconda3/envs/cutie/bin/python
Traceback (most recent call last):
File "/home/tervolab/anaconda3/envs/cutie/bin/torchrun", line 33, in <module>
sys.exit(load_entry_point('torch==2.4.0', 'console_scripts', 'torchrun')())
File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 348, in wrapper
return f(*args, **kwargs)
File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
run(args)
File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
elastic_launch(
File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/tervolab/anaconda3/envs/cutie/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
cutie/train.py FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2024-08-22_21:29:10
host : tervolab-ws1.hhmi.org
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 115342)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2024-08-22_21:29:10
host : tervolab-ws1.hhmi.org
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 115341)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
hkchengrex commented
Thank you for reporting. Fixed.
Note the warning though. You probably want to continue using main training, not pre-training. See https://github.com/hkchengrex/Cutie/blob/main/docs/TRAINING.md#training-command