std::bad_alloc core dump example
jsta opened this issue · 7 comments
I'm trying to extract a raster from the gdb located at:
https://gisdata.mn.gov/dataset/water-lake-bathymetry
- ftp://ftp.gisdata.mn.gov/pub/gdrs/data/pub/us_mn_state_dnr/water_lake_bathymetry/fgdb_water_lake_bathymetry.zip
Running into an error (copied below) that I thought I'd share:
arc_raster.exe data/mn_bathy/water_lake_bathymetry.gdb/ 0 lake_bathymetric_elevation_model
arc_raster.exe data/mn_bathy/water_lake_bathymetry.gdb/ 0 lake_bathymetric_elevation_model
Arc Raster Rescue (url=github.com/r-barnes/ArcRasterRescue, hash=2f4140e9d209355e, compiled=2019-08-23 12:48:01 UTC)
Allocating 4x104576x117376 = 49098850304
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)
Hi @jsta did you end up figuring this out? I don't know anything about C++ but I was assuming that this meant that the file was too large to fit in RAM. Do you know if this is not the issue?
Thanks,
Dave
@jsta: That's a 49GB allocation, so it's not surprising it would fail. Are those dimensions (104576x117376) cells correct?
I'm getting slightly under those numbers but pretty close. It is quite large.
@jsta : Given that the file is only 1.1GB, I'm struggling to see how the dataset itself could be that large.
It's 1.7 GB when extracted as a tif. I don't know if it'll help but here is the gdalinfo readout:
gdalinfo lake_bathymetric_elevation_model.tif
Driver: GTiff/GeoTIFF
Files: lake_bathymetric_elevation_model.tif
Size is 104468, 117233
Coordinate System is:
PROJCRS["NAD_1983_UTM_Zone_15N",
BASEGEOGCRS["NAD83",
DATUM["North American Datum 1983",
ELLIPSOID["GRS 1980",6378137,298.257222101004,
LENGTHUNIT["metre",1]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
ID["EPSG",4269]],
CONVERSION["UTM zone 15N",
METHOD["Transverse Mercator",
ID["EPSG",9807]],
PARAMETER["Latitude of natural origin",0,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8801]],
PARAMETER["Longitude of natural origin",-93,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8802]],
PARAMETER["Scale factor at natural origin",0.9996,
SCALEUNIT["unity",1],
ID["EPSG",8805]],
PARAMETER["False easting",500000,
LENGTHUNIT["metre",1],
ID["EPSG",8806]],
PARAMETER["False northing",0,
LENGTHUNIT["metre",1],
ID["EPSG",8807]]],
CS[Cartesian,2],
AXIS["easting",east,
ORDER[1],
LENGTHUNIT["metre",1]],
AXIS["northing",north,
ORDER[2],
LENGTHUNIT["metre",1]],
ID["EPSG",26915]]
Data axis to CRS axis mapping: 1,2
Origin = (199105.000000000000000,5403520.000000000000000)
Pixel Size = (5.000000000000000,-5.000000000000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( 199105.000, 5403520.000) ( 97d 5'26.65"W, 48d42'43.25"N)
Lower Left ( 199105.000, 4817355.000) ( 96d43' 6.99"W, 43d26'55.03"N)
Upper Right ( 721445.000, 5403520.000) ( 89d59'15.24"W, 48d44'43.05"N)
Lower Right ( 721445.000, 4817355.000) ( 90d15'43.10"W, 43d28'34.80"N)
Center ( 460275.000, 5110437.500) ( 93d30'51.79"W, 46d 8'46.88"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
NoData Value=-3.40282306073709653e+38
Unit Type: metre
Metadata:
RepresentationType=ATHEMATIC
Thanks, @jsta. Does the raster contain large areas of no_data or with pixels that are all the same value? (Does it look like it would compress nicely?)
I'm encountering a similar issue trying to rescue a File Geodatabase raster from https://www.fs.usda.gov/rds/archive/products/RDS-2019-0030/RDS-2019-0030.zip (U.S. Forest Service Riparian Areas layer).
Allocating 1x462083x286682 = 132470878606
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted
This is a national (U.S.) layer, at 10m resolution (though w/ only 2 integer values, and lots of NoData, so the geodatabase is only around 2.15 GB, though an uncompressed version would be much larger. Does ArcRasterRescue have a method for handling rasters that are too large to fit in RAM?