Training failed
shangweichao opened this issue · 9 comments
shangweichao commented
Hello! When I was training your data set, I failed every time to the last step, I changed the batch-size to 2, also failed to the last step, what is your training time? Can you help me out here. Thank you!
lreiher commented
What exactly do you mean by "failed to the last step"? The progress bar shows completed batches per epoch. Once it has completed, validation will run (without an additional progress bar), which will also take a while. Then, the next epoch with a new progress bar will be started. Please post the exact error message!
Depending on the type of model, our training time was ~20-45min/epoch on an RTX 2080 TI.
shangweichao commented
Hello! Thank you very much for your reply. The problems when my program is running are as follows: (1) first of all, I did not modify your file. For the first epoch, the number of images processed is 6639, but when it reaches 6638, there is a problem, as shown in the attached picture (a).
When I changed the batch size to 2, for the first epoch, the number of images processed was 16599, but there was a problem by 16598, as shown in picture (b) in the attachment
…------------------ 原始邮件 ------------------
发件人: "ika-rwth-aachen/Cam2BEV" <notifications@github.com>;
发送时间: 2020年7月13日(星期一) 下午4:19
收件人: "ika-rwth-aachen/Cam2BEV"<Cam2BEV@noreply.github.com>;
抄送: "胖胖的"<2972646161@qq.com>;"Author"<author@noreply.github.com>;
主题: Re: [ika-rwth-aachen/Cam2BEV] Training failed (#1)
What exactly do you mean by "failed to the last step"? The progress bar shows completed batches per epoch. Once it has completed, validation will run (without an additional progress bar), which will also take a while. Then, the next epoch with a new progress bar will be started. Please post the exact error message!
Depending on the type of model, our training time was ~20-45min/epoch on an RTX 2080 TI.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
shangweichao commented
I'm sorry I didn't introduce myself. I'm a student from Xi'an University of technology in Xi'an City, Shaanxi Province, China. My name is Shang Weichao. Now I'm working on the perception of unmanned driving environment.
Hello! For the problem, I got the result that the memory of the server was not enough by querying information and communicating with friends. Then I expanded the memory of the server, but the result is still the same. I am very confused. I don't know if it bothers you. I hope you can help me solve it. If you can share your model for me to learn, I will be very grateful. Thank you again for your reply.
…------------------ 原始邮件 ------------------
发件人: "ika-rwth-aachen/Cam2BEV" <notifications@github.com>;
发送时间: 2020年7月13日(星期一) 下午4:19
收件人: "ika-rwth-aachen/Cam2BEV"<Cam2BEV@noreply.github.com>;
抄送: "胖胖的"<2972646161@qq.com>;"Author"<author@noreply.github.com>;
主题: Re: [ika-rwth-aachen/Cam2BEV] Training failed (#1)
What exactly do you mean by "failed to the last step"? The progress bar shows completed batches per epoch. Once it has completed, validation will run (without an additional progress bar), which will also take a while. Then, the next epoch with a new progress bar will be started. Please post the exact error message!
Depending on the type of model, our training time was ~20-45min/epoch on an RTX 2080 TI.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
shangweichao commented
Hello! I don't know if I can learn your training model. I have tried many times. This problem has puzzled me for a long time. I really have no way. I hope to get your final training results for learning. Thank you very much!
…------------------ 原始邮件 ------------------
发件人: "ika-rwth-aachen/Cam2BEV" <notifications@github.com>;
发送时间: 2020年7月13日(星期一) 下午4:19
收件人: "ika-rwth-aachen/Cam2BEV"<Cam2BEV@noreply.github.com>;
抄送: "胖胖的"<2972646161@qq.com>;"Author"<author@noreply.github.com>;
主题: Re: [ika-rwth-aachen/Cam2BEV] Training failed (#1)
What exactly do you mean by "failed to the last step"? The progress bar shows completed batches per epoch. Once it has completed, validation will run (without an additional progress bar), which will also take a while. Then, the next epoch with a new progress bar will be started. Please post the exact error message!
Depending on the type of model, our training time was ~20-45min/epoch on an RTX 2080 TI.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
lreiher commented
You say you have attached screenshots, but I don't seem to see any attachments.
The number of batches (not images) is 6639 with batch size 5, since there are 33199 training images. If you reduce the batch size to 2 you obviously get 16599 batches.
The fact that the progress bar only reaches 6638/6639 is Keras-related. Under the hood, training phase has already finished and validation is running. As to why your system is running out-of-memory during validation, I cannot help you without further information. The RTX 2080 TI we used has 12GB of GPU memory.
shangweichao commented
Thank you very much for your help in answering my questions. If I need your final model, what conditions do I need? If not, thank you very much! thank you!
…------------------ 原始邮件 ------------------
发件人: "ika-rwth-aachen/Cam2BEV" <notifications@github.com>;
发送时间: 2020年7月13日(星期一) 晚上7:30
收件人: "ika-rwth-aachen/Cam2BEV"<Cam2BEV@noreply.github.com>;
抄送: "胖胖的"<2972646161@qq.com>;"Author"<author@noreply.github.com>;
主题: Re: [ika-rwth-aachen/Cam2BEV] Training failed (#1)
You say you have attached screenshots, but I don't seem to see any attachments.
The number of batches (not images) is 6639 with batch size 5, since there are 33199 training images. If you reduce the batch size to 2 you obviously get 16599 batches.
The fact that the progress bar only reaches 6638/6639 is Keras-related. Under the hood, training phase has already finished and validation is running. As to why your system is running out-of-memory during validation, I cannot help you without further information. The RTX 2080 TI we used has 12GB of GPU memory.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
lreiher commented
I see no need to upload a trained model. All algorithms and data to reproduce our model is available here. I am now closing this issue, thank you for your interest in the project!
shangweichao commented
Thank you very much for your reply. I hope you don't get angry if there is any interruption. Thank you very much!
…------------------ 原始邮件 ------------------
发件人: "ika-rwth-aachen/Cam2BEV" <notifications@github.com>;
发送时间: 2020年7月13日(星期一) 晚上9:04
收件人: "ika-rwth-aachen/Cam2BEV"<Cam2BEV@noreply.github.com>;
抄送: "胖胖的"<2972646161@qq.com>;"Author"<author@noreply.github.com>;
主题: Re: [ika-rwth-aachen/Cam2BEV] Training failed (#1)
I see no need to upload a trained model. All algorithms and data to reproduce our model is available here. I am now closing this issue, thank you for your interest in the project!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
shangweichao commented
Hope not to disturb you, thank you!
…------------------ 原始邮件 ------------------
发件人: "ika-rwth-aachen/Cam2BEV" <notifications@github.com>;
发送时间: 2020年7月13日(星期一) 晚上9:38
收件人: "ika-rwth-aachen/Cam2BEV"<Cam2BEV@noreply.github.com>;
抄送: "胖胖的"<2972646161@qq.com>;"Author"<author@noreply.github.com>;
主题: Re: [ika-rwth-aachen/Cam2BEV] Training failed (#1)
Closed #1.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.