PyEO is designed to provide a set of portable, extensible and modular Python scripts for machine learning in earth observation and GIS, including downloading, preprocessing, creation of baseline composites, classification and validation.
Full documentation is available at https://clcr.github.io/pyeo/build/html/index.html
Example notebooks are available at:
Python library requirements are categorised by Platform (Operating System - OS). For use in the Cloud Processing platform SEPAL - pyeo is already installed in a virtual environment.
SEPAL is a cloud computing platform for geospatial data which offers remote Linux Instances that are customised for performing geospatial analysis in R or Python. More information can be found here: https://github.com/openforis/sepal
.
Package management is performed by Conda, for instructions on how to install Conda, please refer to: https://docs.conda.io/en/latest/.
Note: Conda can be installed as part of Anaconda https://www.anaconda.com/
For installation locally on an OS of your choice, see the sections below.
To install pyeo_1
, put the following commands into Bash (Linux), Terminal (Mac) or the Anaconda Prompt (Windows)
conda install -c conda-forge git
git clone https://github.com/clcr/pyeo_1.git
cd pyeo_1
conda env create --file environment.yml --name pyeo_env
conda activate pyeo_env
python -m pip install -e .
If you do not want to edit pyeo_1
, replace python -m pip install -e .
line with
python -m pip install -vv .
conda install -c conda-forge git
git clone https://github.com/clcr/pyeo_1.git
cd pyeo_1
conda env create --file environment_windows.yml --name pyeo_env
conda activate pyeo_env
python -m pip install -e .
If you do not want to edit pyeo_1
, replace python -m pip install -e .
line with
python -m pip install -vv .
If the OS that pyeo is running on is Windows, we have noticed that pyeo_windows.ini
may need to be saved with ANSI
encoding instead of the usual UTF-8
. See this webpage for more details.
From July 2023, Scihub will be deprecated in favour of the Copernicus Data Space Ecosystem (CDSE). In the meantime, if you wish to download from Scihub, you will need a Scihub account: https://scihub.copernicus.eu/
To use the CDSE, you will need a separate account: https://dataspace.copernicus.eu
Once you have created your account, you will need to enter your email address and password into the credentials_dummy.ini
file in the folder credentials
, like this:
[sent_2]
user=replace_this_with_your_email_address
pass=replace_this_with_your_password
[dataspace]
user=replace_this_with_your_email_address
pass=replace_this_with_your_password
Where user
and pass
under [sent_2]
correspond to your scihub
account details, and user
and pass
under [dataspace]
correspond to your dataspace
account details.
To process Sentinel-2 L1Cs, you will also need Sen2Cor installed: http://step.esa.int/main/third-party-plugins-2/sen2cor/. This installation process is covered in the PyEO_I_Setup.ipynb notebook, available from the notebooks folder.
You can test your installation with by typing the following in Bash/Terminal/Anaconda Prompt:
python
import pyeo_1.classification
or, by running the same import command above, after having started a jupyter notebook via:
jupyter notebook
Please note, if you are using SEPAL, jupyter notebooks have to be started via a GUI method instead of from Bash, see: https://user-images.githubusercontent.com/149204/132491851-5ac0303f-1064-4e12-9627-f34e3f78d880.png
PyEO operates across two stages:
- Across Sentinel-2 (S2) tiles
- Within individual S2 tiles
- Takes a Region of Interest (ROI) and calculates which Sentinel-2 (S2) tiles overlap with the ROI
- Builds a Baseline Composite to compare land cover changes against, by downloading S2 images and calculating the median of these images.
- Downloads images over the Change Period
- Classifies the Composite and the Change images using a classifier in
./models/
- Calculates the change between the from classes and the to classes, for each classified image. This could be changes from forest to bare soil.
- Creates a Change Report describing the consistency of the class changes, highlighting the changes that
PyEO
is confident. - Vectorises the Change Report and removes any changes outside of the ROI
PyEO can be run interactively in the Jupyter Notebooks provided in the Tutorials, but the pipeline method can be run via the Terminal. This process is automated and is suited to the advanced python user.
Both the terminal and notebook methods rely on an a configuration file (e.g. pyeo_linux.ini
, pyeo_windows.ini
, pyeo_sepal.ini
) to make processing decisions.
The below example references pyeo_sepal.ini
, but this can be switched for the Linux or Windows equivalent.
- First, move to where PyEO is installed:
cd pyeo_1
- Now, the pipeline method runs like this. Here we are telling the terminal that we want to invoke
python
to run the scriptrun_acd_national.py
within the folderpyeo_1
, then we pass the absolute path to the initialisation file for your OS. The scriptrun_acd_national.py
requires this path as all the processing parameters are stored in the initialisation file. See below:
python pyeo_1/run_acd_national.py <insert_your_absolute_path_to>/pyeo_sepal.ini
The pipeline uses arguments specified in the .ini
file (short for initialisation), to decide what processes to run.
Here, we will go through the sections of the ini
file and what arguments do what.
[forest_sentinel]
# Acquisition dates for Images in the Change Period, in the form yyyymmdd
start_date=20230101
end_date=20230611
# Acquisition dates for Images for the Baseline Composite, in the form yyyymmdd
composite_start=20220101
composite_end=20221231
# EPSG code, for example - Kenya. This epsg is for areas North of equator and East of 36°E is EPSG:21097
# See https://epsg.io/21097 and https://spatialreference.org/ref/epsg/21097/
epsg=21097
# Cloud cover threshold for imagery to download
cloud_cover=25
# Certainty value above which a pixel is considered a cloud from sen2cor
cloud_certainty_threshold=0
# path to the trained machine learning model for land cover in Kenya
model= ./models/model_36MYE_Unoptimised_20230505_no_haze.pkl
To enable parallel processing of the raster and vector processing pipelines with the do_parallel = True
option enabled in pyeo_sepal.ini
, make the following file an executable by issuing this command:
cd pyeo_1/apps/automation/
chmod u+x automate_launch.sh
Once installation of pyeo_1
is complete, you can follow the tutorial notebooks, which demonstrate the utility of pyeo_1
.
How to Train Your Classifier: https://github.com/clcr/pyeo_1/blob/main/notebooks/PyEO_I_Model_Training.ipynb
Downloading Sentinel-2 Imagery, Creating a Baseline Composite, Performing Automatic Change Detection: https://github.com/clcr/pyeo_1/blob/main/notebooks/PyEO_I_Master_Tutorials.ipynb
Please use the following references when using pyeo:
Roberts, J.F., Mwangi, R., Mukabi, F., Njui, J., Nzioka, K., Ndambiri, J.K., Bispo, P.C., Espirito-Santo, F.D.B., Gou, Y., Johnson, S.C.M. and Louis, V., 2022. Pyeo: A Python package for near-real-time forest cover change detection from Earth observation using machine learning. Computers & Geosciences, 167, p.105192.
Roberts, J., Balzter, H., Gou, Y., Louis, V., Robb, C., 2020. Pyeo: Automated satellite imagery processing. https://doi.org/10.5281/zenodo.3689674
Pacheco-Pascagaza, A.M., Gou, Y., Louis, V., Roberts, J.F., Rodríguez-Veiga, P., da Conceição Bispo, P., Espírito-Santo, F.D., Robb, C., Upton, C., Galindo, G. and Cabrera, E., 2022. Near real-time change detection system using Sentinel-2 and machine learning: A test for Mexican and Colombian forests. Remote Sensing, 14(3), p.707.