Ripples in SpaceTime

Official implementation for: Self-Supervised Spatiotemporal Event Discovery

Code demonstrating the generalization ability of features we obtain from training on interesting change events.

Our change event model is trained on different datasets such as change events, images from Cairo or EuroSat dataset. [Training]

Using these trained models features are extracted for EuroSat images. [Inference]

The generalization is measured using clustering purity. [Evaluation]

Requirements

Pytorch 1.10.0
torchvision 0.11.1
scikit-image 0.19.2
tqdm
PIL
numpy

Pre-trained models

If you do not wish to retrain these models or download our datasets these pre-trained models can be used.

We release 3 pre-trained models

ripplemodel_ours: Our self-supervised model trained using Change Events [Google Drive]
ripplemodel_fulldata: Our model trained on all images from Cairo [Google Drive]
ripplemodel_eurosat: Our model trained on all images from EuroSat dataset [Google Drive]

Dataset

The EuroSat Landcover Classification datasets can be downloaded from their download link.

[Direct Download] [GitHub Repo] (This is required for evaluation.) [90 MB]

We present 2 datasets:

The spatiotemporal RGB images from sentinel-2 for the city of Cairo. [Google Drive]. This data is larger in size [7.4 GB]
Temporal slices of Change Events from the city of Cairo. [Google Drive][205 MB]
This is not needed to download if you are using pre-trained models.

Extract these datasets in the main directory. If not, remember to provide paths to these to in the arguments.

Link for directory with all datasets and models: [Google Drive]

It also contains all features extracted on EuroSat from all the models.

Using the Code

Training

Use training.py to train the models.
Training on EuroSat only requires the EuroSat dataset. python3 training.py -bs 256 -e 20 -mt eurosat -ed PATH/TO/EUROSAT
Other models can be trained similarly.

python3 training.py -bs 256 -e 20 -mt fulldata -fdd PATH/TO/CAIRO
python3 training.py -bs 256 -e 20 -mt ours -fdd PATH/TO/CHANGE_EVENT_SLICES

Inference

Use infer_eurosat.py to obtain features from various models on the EuroSat dataset.
This only required EuroSat dataset.
As stated earlier pre-trained models can be downloaded and used to extract these features

python3 infer_eurosat.py -ed PATH/TO/EUROSAT -mt eurosat # other arguments for -mt (model type) can be imnet, fulldata, ours

This creates a .npz file in the directory eurofeats. The .npz contains features for images from EuroSat that can be used for evaluation.
To get features for a custom dataset same command can be used, if dataset is set like EuroSat (One level subdirectories with images).

Evaluation

For evaluation and visualization we release a jupyter notebook evaluation_and_visualzation.ipynb. Look at notebook to see how use the features from the previous step to perform evaluation.

This table shows the performance of different models in terms of cluster purity evaluated using the notebook.

Method	Average Purity (k=40)
EuroSat	0.518
Fulldata	0.533
ImageNet	0.421
Ours (Change Event Training)	0.542

utkarshmall13/Ripples