IDEA-CCNL/Fengshenbang-LM

KeyError: 'global_step'

TTyb opened this issue · 1 comments

Traceback (most recent call last):
  File "finetune.py", line 194, in <module>
    trainer.fit(model, datamoule, ckpt_path=args.load_ckpt_path)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 696, in fit
    self._call_and_handle_interrupt(
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 735, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1110, in _run
    self._restore_modules_and_callbacks(ckpt_path)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1065, in _restore_modules_and_callbacks
    self._checkpoint_connector.restore_model()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 174, in restore_model
    self.trainer._call_lightning_module_hook("on_load_checkpoint", self._loaded_checkpoint)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1550, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "finetune.py", line 163, in on_load_checkpoint
    global_step_offset = checkpoint["global_step"]
KeyError: 'global_step'

Could you please give more background information ? e.g., version of pytorch lightning and which model is loaded?