This repository provides material used for the paper: <add a link here> Overview ======== The repository contains scripts and Python Notebooks which can be used to perform a comparison of 1) Raster water masks derived from Landsat satellite imagery 2) Vector datasets representing hydrography (OpenStreetMap or Surface Hydrology for Australia) 3) Drainage networks derived from 30m SRTM Some of the scripts make use of Google Earth Engine (GEE). Some of the scripts require https://github.com/gena/ee-runner to run GEE JavaScript API in a command-line. Dirctories ========== bin/ .......................... programs used during conversion gdalcopyprj.cmd ............. copies gographic projection from one raster file to another gdalcopyprj.py osmconvert.exe .............. converts OpenStreetMap pbf binary files to o5m and osm format osmfilter.exe ............... filter OpenStreetMap files config/ ....................... config files use by some of the scripts osmconf.ini ................. used by osmconvert and osmfilter water.osmfilter.params ...... combines tags used to extract hydrography from OpenStreetMap files (polylines and polygons) data/ ......................... input files OpenStreetMap/ SurfaceHydrology/ ........... major rivers extracted from http://www.ga.gov.au grid.kml .................... regular grid used to aggregate results and to parallelize the analysis HydroSHEDS_Catchment_MD.* ... HydroSHEDS cachment boundaries for Murray-Darling River Basin HydroSHEDS_rivers_MD.* ...... HydroSHEDS drainage network graphics/ ..................... images comparing generated water mask and OSM with WOfS and Surface Hydrology <images> notebooks/ .................... Python Notebooks used to perform different steps of the analysis ConvertWaterHistogramsAndThresholds.ipynb ExplodeShpPolylines.ipynb ExplodeShpPolylinesSH.ipynb HAND.ipynb HydroBASINS (fix).ipynb NDWI_thresholds.ipynb OSM water validation using LANDSAT (Murray & Darling).ipynb SHP2KML with fix.ipynb TileCatchmentsToGrid.ipynb UploadToFusionTable.ipynb results/ ...................... some of the results generated during analysis water_and_results_examples/ water.zip authoritative/ .............. screenshots comparing L8 watermask to some of the authoritative datastes src/ javascript/ download_SRTM.js ........... GEE script used to download SRTM data clipped by catchment boundaries download_water.js .......... GEE training.js ................ training dataset, generated within GEE javascript-ee-playground/ ......... GEE Playground scripts OSM_automatic_NDWI.js .......... OSM_compute_buffers.js ......... computes distances between rivers using Goodchilds's method OSM_compute_buffers_export.js .. OSM_stat_per_catchment.js ...... generates aggregated statistics python/ generate-hydro-datasets.py .......... script used to deleniate drainage network and compute HAND generate-hydro-datasets-runner.py ... wraps the above script to avoid out-of-memory probles scripts/ convert.cmd .................... script used to extract hydrography from the OSM files web/ .......................... source code of the Google App Engine website (http://osm-water.appspot.com) httplib2 media oauth2client static app.yaml config_web.py index.html README.md server.py six.py AuthoritativeDatasets.docx ...... early comparisons to the authoritative datasets in Australia Tutorial ======== The overall analysis is performed in a number of steps, starting with the download of all required datasets, such as: * HydroBASINS * OpenStreetMap extracts, relevant for the study area Additional step include preparation of the regular grid file (grid.kml) used to parallelize the analysis. The actual steps required for the analysis are quite time consuming and include: 1. Extraction of the hydrography from the the OSM files (scripts/convert.cmd) Produces two shapefiles (polylines and polygons) representing hydrography 2. Split linear hydrography features into small polylines in order to perform the analysis (notebooks/ExplodeShpPolylines.ipynb) 3. Upload results of 2. to Google Fusion Table (UploadToFusionTable.ipynb) 4. Select required catchment boundaries from HydroBASINS. Catchment size should be smaller than current Google Earth Engine download limit! 5. Upload catchments to the Google Fusion Table 6. Prepare regular grid file (grid.kml) and upload to Google Fusion Table, the resulting asset id needs to be updated in the OSM_automatic_NDWI.js script 7. Download SRTM extracts clipped by catchment boundaries (src/javascript/download_SRTM.js) 8. Deleniate drainage network for every sub-catchment DEM from 6. and compute HAND dataset (and all other related hydrological dataset) using python/generate-hydro-datasets-runner.py 9. Convert results of 7. to regular grid tiles and upload to Google Earth Engine as raster assets. 10. Adjust src/javascript/download_water.js do use the new HAND, grid (a copy of src/javascript-ee-playground/OSM_automatic_NDWI.js). 11. Prepare training set for GEE Playground scripts (10.) by performing script runs for a number of tiles, repeat until the results are good for hilly areas, For flat areas and sufficient cloud-free images the script should produce good results without any changes 12. Detect surface water mask for every grid tile using src/javascript/download_water.js. src/javascript/templates provides example on how to parallelize water detection. 13. Upload (re-tile if necessart) the resulting water mask to Google Earth Engine 14. Compute positional and thematic accuracy using src/javascript-ee-playground/OSM_compute_buffers_export.js and src/javascript-ee-playground/OSM_stat_per_catchment.js