How do you deal the Problem about Memory Error when your patch sets 760000 for CHASE dataset?it 's too big

Question

How do you deal the Problem about Memory Error when your patch sets 760000 for CHASE dataset?it 's too big

BOBKINGS1101 opened this issue 5 years ago · 4 comments

Answer 1 · 2019-11-29T03:25:45.000Z

Hi, without any modification of the code, you can increase system swap size; or you can change the dataloader, so you can load each batch on the fly without pre-storing all in memory.

Answer 2 · 2019-11-29T08:18:36.000Z

Thank you！ How to change the dataloader to load the batch on the fly?

Answer 3 · 2019-11-30T08:51:03.000Z

I trained and test on the STARE dataset. But found the result is not very good (Se very low). Did you train on the STARE dataset?If you do some practice on this Datasets, Could you tell me how many patches you cut in this dataset?

Answer 4 · 2019-12-01T03:09:48.000Z

The same as in the code, very big number of patches. You can first generate those patches and store them on drive, then write a DataSet class to read in batch-by-batch as here https://pytorch.org/tutorials/beginner/data_loading_tutorial.html