Check if raster is in memory
Closed this issue · 8 comments
If I'm not mistaken, we need the raster to be hold in the memory to calculate all metrics. Should we include this into check_landcape()?
I'm not aware that the landscape needs to be in memory. Isn't raster loading the data on-the-fly if it's not in memory?
I need to check that again (didn't have the time yet), but I think because when we transform it to a matrix, it needs to be in memory
We convert all raster data to matrices internally, right? If yes, there should be no necessity to have the rasters in memory. On the contrary, we could in some cases read data from the rasters directly and save some memory and improve performance for large landscapes. What do you think?
I think when converting the raster to a matrix it will be read into the memory once automatically. But I need to look into that further.
When raster determines the file will not fit into memory it reads/writes only the chunks needed from/to disk. It is not necessary to load it to memory. A base R matrix, on the other hand, is in memory, of course.
I think when converting the raster to a matrix it will be read into the memory once automatically. But I need to look into that further.
I also think the same as @mhesselbarth - in most (all?) of the metrics we are converting a raster into a matrix, which reads the whole data into RAM.
I also think the same as @mhesselbarth - in most (all?) of the metrics we are converting a raster into a matrix, which reads the whole data into RAM.
Ah, yes, sorry, I didn't read the comment properly and yes, you're right of course.
What I actually wanted to say is that if we read directly from the raster (from disk) and didn't convert it to matrix first, we could save a lot of memory, because:
(1) I think the conversion is a deep copy -> 2x memory usage
(2) we do not always need complete data in memory. We process the landscapes sequentially and if the RAM is an issue, we could also read the data sequentially into the RAM. Raster does that too and probably we could just use their mechanism.
I will close this for now since the changes this would require are massive!