CGCL-codes/naturalcc

OOM error

SoWhereAreYou opened this issue · 4 comments

Hi,
Thank you before and i find how to train.
But the when I have trained a epoch.It will out of memory afte any set in max_sentences: 1
Was it has not fresh the gpu after ever epoch?
image
Looking for your reply.
Thanks.

We do clear the GPU cache after an epoch.
Please, decrease your validation/eval batch size and try it again.

Thanks for your advice.And i'm sorry for my ignorance. Could you share the memory size of used GPUs and the training time you took?

Hope this may help you.
https://github.com/CGCL-codes/naturalcc/blob/master/run/summarization/neural_transformer/relative/python_wan/python.log

We have many training logs for neural models. Please, check them in the run directory.

Hope this may help you. https://github.com/CGCL-codes/naturalcc/blob/master/run/summarization/neural_transformer/relative/python_wan/python.log

We have many training logs for neural models. Please, check them in the run directory.

I" had read it before.But thank you.