Initialization / data preparation / checkpointing time included?
bignamehyp opened this issue · 2 comments
bignamehyp commented
I assumed the hours field reported in dawn benchmark is between the time each checkpoint is saved and the start of the program. Can we exclude initialization time before training. For example, we can load the entire CIFAR dataset to memory first. And also saving checkpoints to disks is expensive especially when the training is super fast. Can we exclude the checkpointing time as well?
Besides can we report training progress that saved every x epochs?
codyaustun commented
The time to load the data and initialize the model should be included. Checkpointing and calculating validation accuracy doesn't need to be included for v1.0 of DAWNBench.