wv2-processing

Processing scripts for decision-tree land use classification on WorldView-2 images.

Habitat Cover Classes

Below are the habitat classes created by this code. wv_classification_colormap.txt Provides a colormap with similar values and associated colors for use in QGIS (and others).

BA = bare soil
WA = water
DG = degraded mangrove
MA = marsh
SC = scrub/shrub
FW = forested wetland (this is mangrove forest in southwest Florida)
FU = forested upland
UG = upland grass
dev = developed

For the NERRS Mangrove Coast Collective project and related publications the can be consolidated as follows:

Bare/Developed = BA, dev
Water = WA
Degraded mangrove = DG, MA
Healthy mangrove = FW
Upland = FU, UG, SC

Software Dependencies

gdal
pygdal
TODO: more here

Installation

PSC Bridges

$ git clone https://github.com/iceberg-project/Seals.git
# TODO: more here

General Installation

NOTE: if you are running this code on IMaRS's servers (eg userproc or seashell) jump directly to the IMaRS user quickstart document (./docs/imars-local.md).

download: git clone https://github.com/USF-IMARS/wv2-processing.git
install dependencies
1. OS-level:
  - Ubuntu:
    - for gdal: sudo apt istall -y python3-gdal
2. remaining python packages w/ setup.py pip3 install -e .
  - alternatively: pip3 install -r requirements.txt or manually install deps listed therein.

Testing

test data

Test data is stored internally at IMaRS and mounted at /srv/imars-objects/homes/common/wv2-processing/test_data/. To run tests you should create a symlink from there to a dir named test_data in this project root ln -s /srv/imars-objects/homes/common/wv2-processing/test_data/ test_data.

Alternatively, you may download a version of these files from google drive here if you have been granted the appropriate permissions. These files are restricted to IMaRS and collaborators; please do not share them in any form. Once the file is downloaded you must extract this file to wv2-processing/test_data/.

TODO: add PGC's files here & merge directories.

running tests

Python tests herein are generally orchestrated by pytest and live alongside the code they are testing with the suffix _test.

Note that comparing hashes on output files doesn't work well b/c of variations in the script and floating point errors so the tests are not very robust; they mostly just check things run without throwing exception. For much of my testing I had to resort to opening the geotiffs with QGIS and confirming that they look right.

Usage

Overview & Manual Steps

Processing is broken into a few steps. Below is an overview of how each step might be run manually. For specific, detailed examples of running the code see the ./docs/examples/ folder. 0. INPUT_DIR, ORTHO_OUTPUT_DIR, and other variables below must be set (eg INPUT_DIR=/home/tylar/wv_proc/my_input_files).

create resampled tifs using pgc_ortho:
- python ./pgc_ortho.py -p 4326 -c ns -t UInt16 -f GTiff --no-pyramids $INPUT_DIR $ORTHO_OUTPUT_DIR
run the wv_classify script on the resampled tifs
1. the (now outdated) python version: python ./wv_classify.py $ORTH_FILE $ID $MET $CRD $DT $SGW $FILT $STAT $LOC $ID_N $RRS_OUT $CLASS_OUT
2. the MATLAB version: matlab -nodisplay -nodesktop -r "wv_classify('$image2','$input_img_basename','$met','$crd_sys','$dt','$filt','$loc','$SLURM_ARRAY_TASK_ID','$rrs_out','$class_out')"
use gdal or similar tools to mosaic multiple outputs together
- see this gist

Script Parameter Reference

-p                 = projection (4326 is the EPSG code for WGS geographic projection)
-c                 = stretch (ns means "no stretch")
-t                 = output bit depth
-f                 = file format
--no-pyramids      =  prevents the code from creating pyramids in the output GeoTIFF
$INPUT_DIR         = directory for NITF staging/input
$ORTHO_OUTPUT_DIR' = directory for output of pgc_ortho code as GeoTIFF
'$ORTH_FILE        = [this variable is outdated and should be deleted]
$ID                = image file name
$MET               = metadata file
$CRD               = coordinate system (e.g. EPSG 4326)
$DT                = input variable for whether to run the decision tree (DT = 2) or to just run Rrs conversion (DT = 0)
$SGW               = [this variable is outdated and should be deleted]
$FILT              = input variable indicating the size of the moving window filter (1 = 3x2, 2 = 5x5, etc.)
$STAT              = [this variable is outdated and should be deleted]
$LOC               = string identifier (e.g. "NSF_Texas")
$ID_N              = identifier based on the file number being run
$RRS_OUT           = directory for output of Rrs GeoTIFFs
$CLASS_OUT         = directory for output of mapped GeoTIFFs

More usage details in the ./docs directory:

IMaRS local systems use: docs/imars-local.md
SLURM : docs/slurm.md
Airflow: docs/airflow.md

spgriffin/wv2-processing