terra::app does not work when nlyr > 9999

[LaureVelez]

Hi,

I am calculating species richness ( 1 tif by species) using terra::app function. When nlyr argument < or = 9999 is running well with the progress bar.
When nlyr > 9999 it does not work without stopping and the progress bar does not appear.
Do you know if there is a limit in term of number of layers in this function ?

Thanks,

[rhijmans]

The number of layers can be much larger.

x <- rast(nrow=2, ncol=2, nlyr=110000)
values(x) <- 1:size(x)
a <- app(x, mean)

[LaureVelez]

Thanks, here is the dimension of the raster x <- rast(nrow=1804, ncol=3608, nlyr=10463)

I tested it, when nlyr < 9999 is working well when nlyr > 9999 is not working with no error it's just spinning in circles
Do you have any idea ?
Thanks,

[rhijmans]

I made some changes that may have fixed this. Can you try again with the development version? Irrespectively, can you show how you used app, specifically, which function did you apply (argument "fun")?

[robinfreeman]

I'm encoutering a similar, possibly related (?) problem (including when updating to the newest development version - terra 1.7.81)..

dir.create("temp")

for (i in 1:20000) {
  x = rast(nrow=1000, ncol=1000, nlyr=1)
  values(x) = 0
  values(x)[runif(1000, 1, size(x))] = 1 # Filling with a thousand random 1s to make them sparsely populated
  writeRaster(x, datatype = "INT1U", paste0("temp/", i, ".tif"), overwrite = TRUE) # INT1U to keep files small
}

raster_files = list.files('temp', full.names = TRUE)

x = rast(raster_files)

a = app(x, sum)

This results in an 'file does not exist' error, then the R session appears to crash (same occurs if I try to writeRaster the stack as well). Running this in R GUI results in the same problem, but if I then try to recrate the SpatRaster, i get:

temp/1.tif: Too many open files (GDAL error 4)

which makes me wonder if this is an OS limit on number of files open?

Calculating intermediate sums (e.g. in 5000 file chunks) seems to work fine

[rhijmans]

I have added a warning and suggested work-around (split the task or combine the data into fewer files) to the manual. I am not doing more now because this is a tricky matter that I do not understand enough (e.g. the differences between OSs).