Composable data loading based on TorchData

It aims to provide composable Iterable-style and Map-style building blocks called DataPipes that work well out of the box with the PyTorch's DataLoader. It contains functionality to reproduce many different datasets in TorchVision and TorchText, namely including loading, parsing, caching, and several other utilities (e.g. hash checking).

Todo List

Re-organize mroe than 400 datasets in Remote sensing cimmunity in a task-oriented way;
preprocess the datasets in a AI-ready style;
implementing dataset classes for downloading;
supporting for heigh-level repos for specific tasks: obejct detection, segmentation and so forth;
supporting dataloaders in a easy-to-use way for custom projects;
benchmarking cutting-edge CV backbones and models on RS data;
implementing the random sampler for geo-spatial datasets;

Supported datasets:

Continually updating

How to use

Install newest versions of torch and torchdata

sh install_requirements.sh

Install Dataset4EO

git clone git@github.com:DeepAI4EO/Dataset4EO.git
python -m pip install -e .

Then it can be used by

from Dataset4EO.datasets import list_datasets, load, landslide4sense
from torch.utils.data import DataLoader2
from tqdm import tqdm

#list all the supported datasets
print(list_datasets())

#create new dataset object by calling:
datasets_dir = './'
dp = landslide4sense.Landslide4Sense(datasets_dir, split='train')

#Then the corresponding dataset will be downloaded and decompressed automatically

#create a dataloader by calling:
data_loader = DataLoader2(dp.shuffle(), batch_size=4, num_workers=4, shuffle=True, drop_last=True)

#Now, iterating the dataloader for training
for it in tqdm(data_loader):
    print(it)

Add Transformations

from Dataset4EO import transforms

tfs = transforms.Compose(transforms.RandomHorizontalFlip(),
                                 transforms.RandomVerticalFlip(),
                                 transforms.RandomResizedCrop((128, 128), scale=[0.5, 1]))
                                 
ndp = ndp.map(tfs)

Contribution Guidelines

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

Fork the Project
Create your Feature Branch (git checkout -b feature/AmazingFeature)
Commit your Changes (git commit -m 'Add some AmazingFeature')
Push to the Branch (git push origin feature/AmazingFeature)
Open a Pull Request

Citation

@article{earthnets4eo,
    title={EarthNets: Empowering AI in Earth Observation},
    author={Zhitong Xiong, Fahong Zhang, Yi Wang, Yilei Shi, Xiao Xiang Zhu},
    journal = {arXiv:2210.04936},
    year={2022}
}

Acknowledgment

We thank the following open dataset collections:

tanmlh/Dataset4EO