std::length_error exception when running on 9GB DEM raster dataset
hitboxr opened this issue · 7 comments
I'm trying to extract a statewide DEM from here: ftp://ftp.gis.oregon.gov/elevation/DEM/Statewide_Filegeodatabase/OR_DEM_10M.gdb/
I've attached the full log but the relevant part is here:
Opening BaseTable as 'OR_DEM_10M.gdb/a00000027.gdbtable'...
Opening RasterProjection as 'OR_DEM_10M.gdb/a00000027.gdbtable'
Opening BaseTable as 'OR_DEM_10M.gdb/a00000029.gdbtable'...
Allocating 4x65408x48640 = 12725780480
terminate called after throwing an instance of 'std::length_error'
what(): vector::_M_default_append
Aborted
I realize that that's a nearly 13GB allocation and would likely have failed with a bad_alloc like #6 but it seems odd to me that it would throw a length_error before that? Given that this software is for the GIS discipline it should be able to handle any size of data thrown at it unless the physical limits of the machine are incapable of processing it.
Might also be unrelated but I had to add a #include <array> in arc_raster_rescue.hpp to get this to compile as there were two uses of std::array without ever actually defining it.
Might also be unrelated but I had to add a #include in arc_raster_rescue.hpp to get this to compile as there were two uses of std::array without ever actually defining it.
What compiler and version of said compiler are you using?
I acquired your data set, but was unable to reproduce the problem: I'm currently zooming around the GeoTIFF output in QGIS and things look fine.
You might, on your end, try compiling with debug symbols:
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..
make
And then running in GDB:
gdb --args ./arc_raster_rescue.exe ~/temp/arc_raster_error/ftp.gis.oregon.gov/elevation/DEM/Statewide_Filegeodatabase/OR_DEM_10M.gdb/ 0 /z/out
Inside the debugger use:
catch throw
run
The program will halt on the length error. Then type:
bt
to get the backtrace and provide that here.
Might also be unrelated but I had to add a #include in arc_raster_rescue.hpp to get this to compile as there were two uses of std::array without ever actually defining it.
What compiler and version of said compiler are you using?
$ gcc --version
gcc (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Unfortunately catch throw doesn't seem to work here because there's some lovely silent runtime failures in arm linux like below. From what I know this is caused by libraries being compiled with optimizations that the CPU doesn't support, leading to traces like this. I'm not super familiar with gdb or stack debugging though so I could be wrong. For reference this is running on an armhf system.
$ gdb --args ./arc_raster_rescue_debug.exe ../../../Oregon\ Spatial\ Data\ Library\ 10m\ Oregon\ DEM/OR_DEM_10M.gdb/ 0 ./out.tiff
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./arc_raster_rescue_debug.exe...done.
(gdb) catch throw
Catchpoint 1 (throw)
(gdb) run
Starting program: /media/NAS1/GIS/tools/ArcRasterRescue-1.0.0/build/arc_raster_rescue_debug.exe ../../../Oregon\ Spatial\ Data\ Library\ 10m\ Oregon\ DEM/OR_DEM_10M.gdb/ 0 ./out.tiff
Cannot parse expression `.L1185 4@r4'.
warning: Probes-based dynamic linker interface failed.
Reverting to original interface.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Program received signal SIGILL, Illegal instruction.
0xb14faf48 in ?? () from /lib/arm-linux-gnueabihf/libcrypto.so.1.0.0
(gdb) bt
#0 0xb14faf48 in ?? () from /lib/arm-linux-gnueabihf/libcrypto.so.1.0.0
#1 0xb14f88b0 in OPENSSL_cpuid_setup () from /lib/arm-linux-gnueabihf/libcrypto.so.1.0.0
#2 0xb6fe2960 in call_init (l=, argc=argc@entry=4, argv=argv@entry=0xbefff484,
env=env@entry=0xbefff498) at dl-init.c:72
#3 0xb6fe2a3a in call_init (env=, argv=, argc=, l=)
at dl-init.c:30
#4 _dl_init (main_map=0xb6fff958, argc=4, argv=0xbefff484, env=0xbefff498) at dl-init.c:120
#5 0xb6fd7a04 in _dl_start_user () from /lib/ld-linux-armhf.so.3
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Yeah, that stacktrace is not being particularly helpful, unfortunately.
Additional questions:
- Is it possible for you to use a newer compiler and see if the problem magically goes away? GCC 5.4.0 was release on June 3, 2016, almost five years later GCC 10 is the current version. In between, the helpfulness of the error messages and support for C++14, C++17, and C++20 increased considerably.
- How much RAM does your machine have available (in total)?
- Unfortunately not easily, the linux box I was running this on uses a custom ubuntu 16.04LTS distro and I don't want to break anything on that machine.
- It's a 2GB machine.
In related news, I compiled this on a portable Mint x86 distro on my desktop computer and it worked flawlessly. I'm fairly confident that this was a compiler issue given the <array> error I was initially getting with the (very) old compiler I was using. I'm leaving this open for now if you want to dive deep into this, but you're welcome to close this as it was not an error directly associated with the program itself (which worked flawlessly by the way - thank you!).
I'm glad you got it working, and that it was useful to you :-)