[ g e o m o d s ] Geomods is a set of Python modules and console scripts for utilizing elevation data and Digital Elevation Models. ============================================== Python Dependencies: ============================================== - GDAL * - numpy * - scipy - lxml * - requests * - matplotlib ============================================== External Dependencies for DEM generation and data processing: ============================================== - GDAL * - GMT - MBSystem - LASTools - VDatum 4.x * required ============================================== Install via git and pip: ============================================== % git clone https:/github.com/ciresdem/geomods.git % cd geomods % pip install --user --upgrade ./ ============================================== Install via git and setup.py: ============================================== % git clone https://github.com/ciresdem/geomods.git % cd geomods % python setup.py build % sudo python setup.py install --record geomods.install ============================================== Install via zip and setup.py: ============================================== % wget https://github.com/ciresdem/geomods/archive/master.zip % unzip master.zip % cd master % python setup.py build % sudo python setup.py install --record geomods.install ============================================== Uninstall from setup.py: ============================================== % sudo python setup.py clean % sudo xargs rm -r < geomods.install ============================================== Update with setup.py: ============================================== % git pull % sudo python setup.py clean % sudo xargs rm -r < geomods.install % python setup.py build % sudo python setup.py install --record geomods.install ============================================== Python Modules: ============================================== - fetches - Data fetching - waffles - DEM generation and processing - datalists - datalist processing - regions - region processing - xyzfun - xyz data functions - gdalfun - gdal functions - gmtfun - gmt functions - mbsfun - mb-system functions - vdatumfun - vdatum functions - utils - general utilities and processing ============================================== Console Scripts: ============================================== - waffles - Generate a Digital Elevation Model and derivatives. - fetches - Fetch elevation data from various sources. - datalists - process and analyze datalists. - gdal_chunk.py - Split a gdal grid into chunks. - gdal_crop.py - Crop a gdal grid by it's nodata value. - gdal_clip.py - Clip a gdal grid to an ogr vector. - gdal_null.py - Generate a null gdal grid. - gdal_mask.py - Apply a raster mask to a gdal grid. - gdal_split.py - Split a gdal grid based on z value. - gdal_query.py - Query the z value of a gdal grid with x/y coordinates. - vdatum_cmd.py - Use NOAA's VDatum via command-line. - hillshade.sh - Generate an ETOPO1 style hillshade image. ============================================== Examples: ============================================== - fetch and process lidar data from NOAA's Digital Coast for all the regions found in tiles.shp % fetches dc -R tiles.shp -p --where 'Datatype LIKE "lidar%"' - list all the NOS hydrographic data urls in the given region % fetches nos -R -90/-89/29/30 -l > nos.urls - generate a DEM for each region found in tiles.shp using GMT surface and the data found in input_data.datalist % waffles -M surface -R tiles.shp -E .00000925925 input_data.datalist --verbose - generate spatial-metadata from the datalist input_data.datalist % waffles -R tiles.shp -E .00000925925 input_data.datalist --verbose -M spat-meta - generate a vertical datum conversion grid: % waffles -R -90/-89/29/30 -E 0.00083333 -M vdatum:ivert=navd88:o_vert=mhw - generate a 'bathymetry surface' DEM: % waffles -R region.shp -E 1s:.3333333s -X6 -M surface:upper_limit=0 -C coast_ply.shp:invert=True - generate a nearest-neighbor DEM using all the .tif grid files in the current directory: % waffles -R region.shp -E1s -Mnearneighbor:radius=3s *.tif - split grid file tmp.tif into chunks of 1000x1000 cells: % gdal_chunk.py tmp.tif -c 1000 - transform the vertical datum of an xyz data file: % vdatum_cmd.py elev_data.xyz -i lmsl:ft:sounding -o navd88:ft:height ============================================== Waffles CLI: ============================================== waffles [OPTIONS] <datalist/entry> Generate DEMs and derivatives and process datalists. General Options: -R, --region Specifies the desired REGION; This can either be a GMT-style region ( -R xmin/xmax/ymin/ymax ) or an OGR-compatible vector file with regional polygons. If a vector file is supplied it will search each region found therein. If omitted, use the region gathered from the data in DATALIST. -E, --increment Gridding CELL-SIZE in native units or GMT-style increments. append :<inc> to resample the output to the given <inc>: -E.3333333s:.1111111s -F, --format Output grid FORMAT. [GTiff] -M, --module Desired DEM MODULE and options. (see available Modules below) syntax is -M module:mod_opt=mod_val:mod_opt1=mod_val1:... -O, --output-name BASENAME for all outputs. -P, --epsg Horizontal projection of data as EPSG code [4326] -X, --extend Number of cells with which to EXTEND the REGION. append :<num> to extend the processing region: -X6:12 -T, --filter FILTER the output DEM using one or multiple filters. <fltr:fltr_val:split_value=z> Available filters: 1: perform a Gaussian filter at -T1:<factor>. 2: use a Cosine Arch Filter at -T2:<dist(km)> search distance. 3: Spike Filter at -T3:<stand-dev. threshhold>. The -T switch may be set multiple times to perform multiple filters. Append :split_value=<num> to only filter values below z-value <num>. e.g. -T1:10:split_value=0 to smooth bathymetry using Gaussian filter -Z, --z-region Restrict data processing to records that fall within the z-region Use '-' to indicate no bounding range; e.g. -Z-/0 will restrict processing to data records whose z value is below zero. -C, --clip CLIP the output to the clip polygon -C<clip_ply.shp:invert=False> -K, --chunk Process the region in CHUNKs. -K<chunk-level> -W, --w-region Restrict data processing to records that fall within the w-region (weight). Use '-' to indicate no bounding range; e.g. -W1/- will restrict processing to data records whose weight value is at least 1. -G, --wg-config A waffles config JSON file. If supplied, will overwrite all other options. generate a waffles_config JSON file using the --config flag. -p, --prefix Set BASENAME to PREFIX (append inc/region/year info to output BASENAME). -r, --grid-node Use grid-node registration, default is pixel-node -w, --weights Use weights provided in the datalist to weight overlapping data. -a, --archive Archive the datalist to the given region. -m, --mask Generate a data mask raster. -u, --uncert Generate an associated uncertainty grid. -c, --continue Don't clobber existing files. -q, --quiet Lower verbosity to a quiet. (overrides --verbose) --help Print the usage text --config Save the waffles config JSON and major datalist --modules Disply the module descriptions and usage --version Print the version information --verbose Increase the verbosity Datalists and data formats: A datalist is a file that contains a number of datalist entries, while an entry is a space-delineated line: `/path/to/data format weight data,meta,data` Supported datalist formats: -1 ['datalist', 'mb-1'] -2 ['zip', 'gz'] -4 ['nos', 'dc', 'gmrt', 'srtm_cgiar', 'srtm_plus', 'mar_grav', 'charts', 'mb', 'tnm', 'emodnet', 'chs', 'hrdem', 'cudem'] 168 ['xyz', 'csv', 'dat', 'ascii'] 200 ['tif', 'img', 'grd', 'nc', 'vrt', 'bag'] 300 ['las', 'laz'] Modules (see waffles --modules <module-name> for more info): surface, triangulate, cudem, nearest, num, vdatum, mbgrid, invdst, average, linear, spat-meta, coastline, uncertainty ============================================== Fetches CLI: ============================================== fetches [OPTIONS] <module[:parameter=value]* ...> Fetch geographic elevation data. General Options: -R, --region Specifies the desired region to search; This can either be a GMT-style region ( -R xmin/xmax/ymin/ymax ) or an OGR-compatible vector file with regional polygons. If a vector file is supplied it will search each region found therein. -W, --where restricted_where: Attribute query (like SQL WHERE) -E, --increment Blockmedian/mean CELL-SIZE in native units or GMT-style increments. -Z, --z-region Restrict data processing to records that fall within the z-region Use '-' to indicate no bounding range; e.g. -Z-/0 will restrict processing to data records whose z value is below zero. -F, --fg-config A fetches config JSON file. If supplied, will overwrite all other options. generate a fetches_config JSON file using the --config flag. -l, --list Return a list of fetch URLs in the given region. -d, --dump Dump the XYZ elevation data in WGS84 to stdout. -p, --process Process fetched elevation data to ASCII XYZ format in WGS84. If -E or -Z are set, processing will use those switches in data processing. -u, --update Update the Fetches Remote Elevation Datalist. -i, --index Print the fetch FRED results in the given region. --help Print the usage text --config Save the fetches config JSON --version Print the version information Modules (see fetches --modules <module-name> for more info): dc nos charts ncei_thredds usace tnm gmrt mb mar_grav srtm_plus emodnet ngs chs hrdem Examples: % fetches -R -90.75/-88.1/28.7/31.25 nos --where "Date > 2000" % fetches -R region.shp -p dc nos:datatype=bag charts:datatype=enc % fetches -R region.shp dc:datatype=lidar -l > dc_lidar.urls % fetches -R -89.75/-89.5/30.25/30.5 tnm:ds=4:formats=IMG gmrt:res=max:fmt=geotiff