pfnet/pytorch-pfn-extras

Add manager._start_time to state_dict

linshokaku opened this issue · 1 comments

if self._start_time is None:
self._start_time = _get_time()
self.start_extensions()

@property
def elapsed_time(self) -> float:
if self._start_time is None:
raise RuntimeError(
"Unavailable until the initial run_iteration call."
)
return _get_time() - self._start_time

When restarting from a snapshot, _start_time is initialized and elapsedtime cannot be obtained correctly, so _start_time should be treated as state.

If we are to fix this, I think it's better to record elapsed_time in state dict and accumulate the "real" time consumed (i.e., excluding the time process is stopped)