/awesome-earthobservation-code

curated list of awesome tools, tutorials, code, helpful projects, links, stuff about Earth Observation and Geospatial stuff!

Primary LanguageHTMLCreative Commons Zero v1.0 UniversalCC0-1.0

Awesome-EarthObservation-Code

A curated list of awesome tools, tutorials, code, helpful projects, links, stuff about Earth Observation and Geospatial stuff!

This list was started based on #scenefromabove podcast lunchtime discussions

This is being extended frequently in May 2020. Please note that this is not offically an awesome list (yet). Please help me to get it there by contributing and commenting. guidelines

Annotations are based on the headers - where available - on the github accounts

Scene From Above Podcast

Alastair Graham @ajggeoger and Andrew Cutts @map_andrew come together to present an informal podcast @eoscenefrom looking at the world of modern remote sensing and EO. Fuelled by their passion for all things raster and geospatial, the #scenefromabove podcast aims to be a mix of news, opinion, discussion and interviews.

Contents

Shortcuts

Start Here

OpenEO covers many of the bases, hard to know whether to break it into different categories, it has many components. At present I mention it here at the start only.

  • Open EO - openEO develops an open API to connect R, Python, JavaScript and other clients to big Earth observation cloud back-ends in a simple and unified way.

Python processing of imagery non deep learning

  • StarFM for Python - The STARFM fusion model for Python (image fusion)
  • Python from space - Python Examples for Remote Sensing
  • count blue pixels - This project is an experiment in using simple image processing techniques on satellite images downloaded from Google Maps in order to quantify the relative density of temporary shelters in adjacent qudarants. Python Ruby
  • Satellite imagery analysis with Python - Getting acquainted with the concept of satellite imagery data and how it can be analyzed to investigate real-world environmental and humanitarian challenges. Python Jupyter Notebooks
  • Povetry predition using satellite imagery - Poverty Prediction by Combination of Satellite Imagery
  • Remote Sensing indicies calc - Calculate spectral remote sensing indices from satellite imagery
  • Satellite imagery in Pakistan - This repository contains a study how we can examine the vegetation cover of a region with the help of satellite data. The notebook in this repository aims to familiarise with the concept of satellite imagery data and how it can be analyzed to investigate real-world environmental and humanitarian challenges.
  • s3 tools - A collection of sentinel 3 processing tools
  • eumetsat -python - Shows how to read and plot satellite data from EUMETSAT NETCDF files
  • unidata on GOES-16 - This notebook shows how to make a true color image from the GOES-16 Advanced Baseline Imager (ABI) level 2 data. We will plot the image with matplotlib and Cartopy.
  • esa_sentinel - ESA Sentinel Search & Download API
  • MetPy - MetPy is a collection of tools in Python for reading, visualizing and performing calculations with weather data.
  • EarthPy - A package built to support working with spatial data using open source python. docs
  • RasterFrames / pyrasterframes - brings together Earth-observation (EO) data access, cloud computing, and DataFrame-based data science. docs
  • aqua-monitor - Monitoring surface water changes from space at global scale. Also checkout the app
  • Ocean Color - Modis - introduction to accessing and plotting ocean color satellite data from MODIS
  • SIF tools - some tools for accessing OCO-2 data
  • SIAC - A sensor invariant Atmospheric Correction (SIAC) alg doc
  • S2_TOA_TO_LAI - From Sentinel 2 TOA reflectance to LAI
  • cresi - Road network extraction from satellite imagery, with speed and travel time estmates
  • COG Validator - Cloud Optimized GeoTIFF validation service
  • 6S_emulator - Atmospheric correction in Python using a 6S emulator
  • bv - Quickly view satellite imagery, hyperspectral imagery, and machine learning image outputs directly in your iTerm2 terminal. Python
  • mapchete - Tile-based geodata processing using rasterio & Fiona Python
  • Climate data science - Climate Data Science and Earth Observation with Python
  • unmixing - Interactive tools for spectral mixture analysis of multispectral raster data in Python

Resources for R

R is not my area of expertise so this section is lighter than I'd like, plus I'd love to know what is a useful resource

  • R-Spatial - This book provides a short introduction to satellite data analysis with R.
  • GDAL Cubes - Earth Observation Data Cubes from Satellite Image Collections. Also here on github
  • Image Classification with RandomForests in R - The goal of this post is to demonstrate the ability of R to classify multispectral imagery using RandomForests algorithms.
  • R code for ML in Sat imagery - # Random Forest image classification Adapted from stackoverflow.
  • whiteboxR - An R frontend of the advanced geospatial data analysis platform - whitebox-tools.
  • RasterVIS - Methods for enhanced visualization and interaction with raster data. It implements visualization methods for quantitative data and categorical data, both for univariate and multivariate rasters. It also provides methods to display spatiotemporal rasters, and vector fields.
  • Landsat - Processing of Landsat or other multispectral satellite imagery. Includes relative normalization, image-based radiometric correction, and topographic correction options.
  • rnoaa - R interface to many NOAA data APIs
  • MODISTools - Interface to the MODIS Land Products Subsets Web Services Docs
  • A Step-by-Step Guide to Making 3D Maps with Satellite Imagery in R - Walk you through [on] how to obtain the data required to make these types of maps, as well as the R code used to generate them
  • landsatlinkr - An automated system for creating spectrally consistent and cloud-free Landsat image time series stacks from a combination of MSS, TM, ETM+, and OLI sensors project

Languages other than Python

  • Georust - A collection of geospatial tools and libraries written in Rust
  • ArchGDAL - Julia - Julia A high level API for GDAL - Geospatial Data Abstract
  • GeoTrellis homepage - GeoTrellis is a geographic data processing engine for high performance applications. Scala
  • GDAL with GoLang - Go (golang) wrapper for GDAL, the Geospatial Data Abstraction Library
  • C++ gdalcubes - Earth observation data cubes from GDAL image collections C++
  • RSGLib - The remote sensing and GIS software library (RSGISLib) is a set of C++ libraries and commands for the processing of spatial data (raster and vector). Functionality is via Python interface though
  • WhiteBox with Java - An open-source GIS and remote sensing package - Java
  • Perl extension for GDAL - Geo::GDAL - Perl extension for the GDAL library for geospatial data
  • PDAL - PDAL is Point Data Abstraction Library. GDAL for point cloud data.
  • force - Framework for Operational Radiometric Correction for Environmental monitoring in c
  • LLR-landTrendr - Landsat-based Detection of Trends in Disturbance and Recovery algorimth modified to accept LandsatLinkr-processed imagery. IDL

Training and learning

  • Earth Data Lab - A site dedicated to tutorials, course and other learning materials and resources developed by the Earth Lab team
  • EO College Github
  • profLewis-geog0111 - UCL Geography: 4th year course, Scientific Computing
  • Intro to Geospatial Vector and Raster - Data Carpentry’s aim is to teach researchers basic concepts, skills, and tools for working with data so that they can get more done in less time, and with less pain.
  • Andrew Cutts Github - I am an Earth Observation and Geospatial enthusiast, primarily using Python to automate and process images at scale using computer vision
  • Open Geo Tutorial - Tutorial of fundamental remote sensing and GIS methodologies using open source software in Python
  • Foss4gUKJupyter - FOSS4G UK 2019 Workshop "Geoprocessing with Jupyter Notebooks"
  • Geoprocessing with Python - GIS circa 2009 - This material is really old and some of it is outdated (not all, though!). One of these days I might get around to putting newer class materials online, but you're stuck with this for now.

Deep learning and Machine Learning

Curated lists

  • Christoph Rieke git hub for much much more
  • Deep Vector - A curated list of resources focused on Machine Learning in Geospatial Data Science.
  • Robin Cole on satellite imagery and deep learning resources - Resources for deep learning with satellite & aerial imagery Specific examples
  • TernausNetV2 - TernausNetV2: Fully Convolutional Network for Instance Segmentation [paper](TernausNetV2: Fully Convolutional Network for Instance Segmentation)
  • CNN-Sentinel -Analyzing Sentinel-2 satellite data in Python with Keras (repository of our talks at Minds Mastering Machines 2019 and PyCon 2018)
  • Image patches - Using a U-Net for image segmentation, blending predicted patches smoothly is a must to please the human eye.
  • Fast AI Satellite imagery resources
  • Crop yield prediction - The motive here is to predict the yield of crops of a particular farm by the change in pixels of the image of farm yearly. Uses Tensorflow
  • Houston Flooding with deep learning - Using A Segmentation Neural Net to map out flooded areas of Houston TX using satellite imagery
  • Satellite Imagery Classification with R - Pixel based classification of satellite imagery - feature generation using Orfeo Toolbox, feature selection using Learning Vector Quantization, CLassification using Decision Tree, Neural Networks, Random Forests, KNN and Naive Bayes Classifier
  • SpaceNet building detection - Project to train/test convolutional neural networks to extract buildings from SpaceNet satellite imageries.
  • Road segmentation - Road Detection in satellite imagery. Semantic segmentation is the process of classifying each pixel of an image into distinct classes using deep learning. This aids in identifying regions in an image where certain objects reside.This aim of this project is to identify and segment roads in aerial imagery. Detecting roads can be an important factor in predicting further development of cities, and this concept plays a major role in GeoArchitect (A project which I started). Segmentation of roads is important to map-based applications and is used for finding distances or shortest routes between two places.
  • Super resolution (srcnn) - Super Resolution for Satellite Imagery
  • Pixel decoder - A tool of running deep learning algorithms for semantic segmentation with satellite imagery
  • Detecting ships - Using Satellite Imagery to detect ships (Basic Object Detection)
  • deepOSM - Train a deep learning net with OpenStreetMap features and satellite imagery.
  • Keras for computer vision (Maxime Lenormand GitHub) - Introductions to Keras to perform computer vision tasks, with data exploration, error analysis and improving results.
  • Airplane image classification - This article details building a ML pipeline to classify the presence of planes in satellite images using a Convolutional Neural Network (CNN).
  • TorchSat - an open-source deep learning framework for satellite imagery analysis based on PyTorch. Python docs
  • ml_drought - Machine learning to better predict and understand drought Python. docs
  • pycrop yield prediction - A PyTorch Implementation of Jiaxuan You's Deep Gaussian Process for Crop Yield Prediction Python
  • neat-EO - Efficient AI4EO OpenSource framework Python

Great Github accounts with example projects where possible

Please do explore these accounts, there are some absolutely brilliant projects on these accounts, some have more than I have shown here!

GDAL of course

  • GDAL Cheat Sheet - Cheat sheet for GDAL/OGR command-line tools
  • GDAL / OGR cookbook - This cookbook has simple code snippets on how to use the Python GDAL/OGR API
  • GDAL tutorial - This blogpost gives in an introduction to GDAL/OGR and explains how the various command line tools can be used.
  • Evan Roualt github - Free and open source geospatial developper, extensively supports GDAL

Earth Observation coding on YouTube

(presenters listed where possible)
There are many videos relating to Earth Observation and coding, especially Python. This is really such a small collection of videos here. I have attempted to only include ones with good audio and code examples.

  • xArray at PyConUK2018 - Robin Wilson - Processing thousands of satellite images to understand air quality in the UK - it's efficient and easy with XArray
  • Visualizing & Analyzing Earth Science Data Using PyViz & PyData - Julia Signell - In this talk, we'll work through some specific workflows and explore how various tools - such as Intake, Dask, Xarray, and Datashader - can be used to effectively analyze and visualize these data. Working from within the notebook, we'll iteratively build a product that is interactive, scalable, and deployable.
  • Hands on Satellite Imagery 2019 edition - Sara Safavi - In this tutorial, gain hands-on experience exploring Planet’s publicly-available satellite imagery and using Python tools for geospatial and time-series analysis of medium- and high-resolution imagery data. Using free & open source libraries, learn how to perform foundational imagery analysis techniques and apply these techniques to real satellite data.
  • Python from space - Katherine Scott - In this talk we will work through a jupyter notebook that covers the satellite data ecosystem and the python tools that can be used to sift through and analyze that data. Topics include python tools for using Open Street Maps data, the Geospatial Data Abstraction Library (GDAL), and OpenCV and NumPy for image processing.
  • Remote Sening with Python in Jupyter - In this video we're looking at using Google Earth Engine in Jupyter with the Python API.
  • Writing Image Processing Algorithms with ArcGIS/ArcPy - Jamie Drisdelle - learn how your algorithms can integrate with the raster processing and visualization pipelines in ArcGIS. We’ll demonstrate the concept and discuss the API by diving deep into a few interesting examples with a special focus on multidimensional scientific rasters.
  • Google Earth Engine Python - Qiusheng Wu - Introducing the geemap Python package for interactive mapping with Google Earth Engine and ipyleaflet.
  • Google Earth Engine EE101 Condensed - Noel Gorelick - Introduction to the Earth Engine API and a conceptual overview of key functionality such as compositing, reducing, mapping, zonal statistics and cluminating with building a small app.
  • Image classification with RandomForests using the R languageIn this video I show how to import a Landsat image into R and how to extract pixel data to train and fit a RandomForests model. I also explain how to conduct image classification and how to speed it up through parallel processing.
  • GeoPython 2019 stream - 17:23 Machine Learning for Land Use/Landcover Statistics of Switzerland (Adrian Meyer), 50:58 How to structure geodata, 1:18:13 Terrain segmentation with label bootstrapping for lidar datasets, case of doline detection (Rok Mihevc), 2:34:41 Bias in machine learning, 3:06:23 Software for planning research aircraft missions (Reimar Bauer), 3:32:38 How Technology Moves Fast (PJ Hagerty) , 5:02:05 Spotting Sharks with the TensorFlow Object Detection API (Andrew Carter), 5:40:23 Center for Open Source Data and AI Technologies (CODAIT), 6:03:40 Bayesian modeling with spatial data using PyMC3 (Shreya Khurana) (Sound at 6:04:23 ^^), 7:02:45 Understanding and Implementing Generative Adversarial Networks(GANs) (Anmol Krishan Sachdeva), 7:37:00 Messaging with Satellites from Anywhere on the Planet (Andrew Carter), 8:04:52 Automation of the definition and optimizatino of census sampling areas using AREA (GRID3) (Freja Hunt), 8:35:26 Coastline Mapping with Python, Satellite Imagery and Computer Vision (Rachel Keay)

Earth Engine

EO Geospatial companies or orgs making big contributions

Github accounts only with examples of work. Also contains (Python) libraries for processing satellite data. This list aims at highlighting the great work some of the companies / organisations are doing or have done that contribute to the bigger ecosystem. Examples of the accounts work is not exhaustive, some do much more than shown below:

  • development seed
    • Landsat-Util - A utility to search, download and process Landsat 8 satellite imagery
    • GeoLamda - Create and deploy Geospatial AWS Lambda functions
    • LabelMaker - Data Preparation for Satellite Machine Learning docs
  • mapbox
    • rasterio - Rasterio reads and writes geospatial raster datasets
    • Robosat - Semantic segmentation on aerial and satellite imagery. Extracts features such as: buildings, parking lots, roads, water, clouds
  • Planet Labs, now just Planet
  • Digital Globe - now Maxar
  • Azavea
  • Radiant Earth foundation
    • STAC Spec - SpatioTemporal Asset Catalog specification - making geospatial assets openly searchable and crawlable
  • Sentinel Hub
    • EO Learn - Earth observation processing framework for machine learning in Python
    • EO Browser Custom Scripts - A repository of custom scripts to be used with Sentinel Hub
    • EO flow - Collection of TensorFlow 2.0 code for Earth Observation applications
    • SentinelHub-py - Download and process satellite imagery in Python using Sentinel Hub services.
  • Opendatacube
  • PyTroll
    • SatPy - Python package for earth-observing satellite data processing
    • pyresample - Geospatial image resampling in Python
  • CosmiQ
    • Solaris - CosmiQ Works Geospatial Machine Learning Analysis Toolkit
    • CometTS - Comet Time Series Toolset for working with a time-series of remote sensing imagery and user defined polygons
    • SpaceNet6 Baseline - Baseline algorithm for the SpaceNet 6 Challenge
  • Theia software and tools
  • sparkgeo
  • Geoscience Australia
    • PyRate - A Python tool for estimating velocity and time-series from Interferometric Synthetic Aperture Radar (InSAR) data.
    • Digital Earth Australia Notebooks - R- Jupyter Notebooks, tools and workflows for continental-scale earth observation/geospatial analysis with Open Data Cube (link above) and xarray
  • Dymaxion Labs
    • dask-rasterio - Read and write rasters in parallel using Rasterio and Dask
    • ap-latem - Detection of slums and informal settlements from satellite imagery
  • Satellogic
    • Telluric - telluric is a Python library to manage vector and raster geospatial data in an interactive and easy way
    • Orbit predictor - Python library to propagate satellite orbits.
  • senbox-org - SNAP - ESA's SentiNel Application Platform
    • s3tbx - A toolbox for the OLCI and SLSTR instruments on board of ESA's Sentinel-3 satellite - Java
    • s2tbx - Sentinel 2 Toolbox (s2tbx) - Java
    • s1tbx - The Sentinel-1 Toolbox - Java
  • Nasa-gibs - for info here
    • onearth - High-performance web services for tiled raster imagery and vector tiles
    • Worldview - Interactive interface for browsing global, full-resolution satellite imagery Javascript application here
    • mrf - GDAL-compatible file format driver designed for fast access to imagery
  • mundialis
    • actinia core - Actinia Core is an open source REST API for scalable, distributed, high performance processing of geographical data that uses mainly GRASS GIS for computational tasks.
    • actinia plugin - This actinia plugin is designed for efficient satellite data handling, especially Landsat and Sentinel-2 scenes

QGIS

DEM projects

  • Tin Terrain - A command-line tool for converting heightmaps in GeoTIFF format into tiled optimized meshes.
  • TauDEM - Terrain Analysis Using Digital Elevation Models (TauDEM) software for hydrologic terrain analysis and channel network extraction. Docs
  • DEM.net - Digital Elevation model library in C#. 3D terrain models, line/point Elevations, intervisibility reports. Docs
  • Stereo Mapping to create Elevation with Python - Satellite Stereo Pipeline

SAR

  • SAR docker - Source files for Docker image mort/sardocker/
  • awesome SAR - A curated list of awesome Synthetic Aperture Radar (SAR) software, libraries, and resources.

LiDAR

  • pyGEDI - pyGEDI is a Python Package for NASA's Global Ecosystem Dynamics Investigation (GEDI) mission, data extraction, analysis, processing and visualization.
  • GEDI extraction script - Python script to take GEDI level 2 data and convert variables to a geospatial vector format
  • rGEDI - rGEDI: An R Package for NASA's Global Ecosystem Dynamics Investigation (GEDI) Data Visualization and Processing.
  • ICESAT extraction script - Python script to convert from ICESat-2 ATL08 HDF data to shapefile. Usage: 'python icesat2_shp.py
  • ICESAT tools - Tools and code for Icesat-2 data analysis (Python)
  • usgs-lidar - AWS Entwine Point Tiles USGS LiDAR Public Dataset GitHub repo

InSAR

  • ISCE - InSAR Scientific Computing Environment version 3 alpha
  • LiCSBAS - LiCSBAS package to carry out InSAR time series analysis using LiCSAR products
  • MintPy - Miami InSAR time-series software in Python
  • Pyrocko - Can be utilized flexibly for a variety of geophysical tasks, like seismological data processing and analysis, modelling of InSAR, GPS data and dynamic waveforms, or for seismic source characterization.
  • InSARFlow - Parallel InSAR processing for Time-series analysis

Visualisation

Regular blogs of significant interest or posts of interest

EO code Competitions

Useful EO code based twitter accounts

  • pyGEDI - pyGEDI is a Python Package for NASA's Global Ecosystem Dynamics Investigation (GEDI) mission, data extraction, analysis, processing and visualization.

Interesting Non EO parts Python

This bit could potentially become the most valuable resource. Lets not ignore other sectors/industries/data science, instead lets embrace it and learn from all that other amazing stuff!

Interesting Non EO parts other languages

This section is aimed more a data science/programming resources that 'might' be applicable to Earth Observation, that are not Python

A footnote on awesome

There are many awesome lists relating to 'Geo'. I use that term as widely as possible. This list is not meant to replace these lists. Earth Observation is still way behind the GIS world in terms of audience, reach, number of users etc. Things are changing though, by bringing these links together I hope you can see that there has been so much progress in the last 5 years. I do hope these links are helpful espcially to those who are new to Earth Observation, but also to people like me who with several years of experience think they may have seen it all - we haven't and there is still so much to learn. Earth Observation is not just an academic 'thing' or a basemap anymore, it forms the basis for a growing and diverse business environment. Lets embrace this.

Finally, I wanted to acknowledge a couple of awesome Earth Observation lists that you may list to check out:

End