fort13 builder crashing with large landcover datasets
jlwoodr opened this issue · 11 comments
Hey Zach,
I was testing out the fort.13 creator and I was having problems with the code failing on larger datasets (16654 X 12031 pixels). I then tried it with a clipped portion of the larger dataset (3930 X 3784 pixels) and it worked fine. Any thoughts to why this would happen? Does docker have a memory limit? The code starts out fine with the larger data, but then dies at about 15%. Thanks!
Can you try doing:
Griddata_obj.setRasterInMemory(False)
This should minimize memory usage. I'm not sure if there is an explicit limit except that on your machine, but best to make it not read the whole thing into memory anyway for large datasets. For small datasets, there is a performance gain to have it in memory directly.
Still fails at the same spot with the larger dataset. I am using the "Average" interpolation. Could that be the problem? Is there a nearest neighbor option?
Nope, that crashes too at the same node number.
Would it be possible for you to share your script and inputs to my TWI email?
Yep, sure can. Thanks!
Okay, so I think I am running out of memory on these datasets. I checked the task manager and I am maxing out cpu and memory on my desktop. I am going to tile them in to 4s and see if that works.
Seems to be working fine now. Also, I am currently just looping through a list of landcover rasters. Is there a way to simply add this list to the interpolator instead of having to loop through every node for every raster?
And actually I do not think the looping is working. It seems to be totally rewriting the fort.13 file each time and overwriting the nodal attributes in vertices covered by previous rasters.
This issue seems to have been resolved. The issues were:
- The raster used for directional wind reduction must use a meters based coordinate system
- Using the method setInterpolationFlags forces the re-interpolation of all points. Instead, the setInterpolationFlag method can be used to select specific locations
Awesome, I will make the appropriate changes. Thank you!
Anytime. I sent some additional info to your email