Quick and simple memmap for cases with a lot of images
FlorentLM opened this issue · 0 comments
FlorentLM commented
Hey,
I was trying this on a dataset with about 350 images and masks, and my system did not have ebough memory to load everything at once.
So I added a simple check on RAM and VRAM, and if the dataset is too big it'll use numpy's memmap feature. It could prob be made a bit nicer with other dependencies (psutil and zarr for instance) but this current implementation does not require any other dependencies
Feel free to have a look
(it's currently running and all seems fine ...but I'll update if any issue surfaces later down the road!)