fulmo

Template to start your deep learning project based on PyTorchLightning for rapid prototyping.

Contents

fulmo

Why Lightning + Hydra + Albumentations?

PyTorch Lightning provides great abstractions for well structured ML code and advanced features like checkpointing, gradient accumulation, distributed training, etc.
Hydra provides convenient way to manage experiment configurations and advanced features like overriding any config parameter from command line, scheduling execution of many runs, etc.
Albumentations (Optional) provides many image augmentation. Albumentations supports all common computer vision tasks such as classification, semantic segmentation, instance segmentation, object detection, and pose estimation.

Features

Pipelines based on hydra-core configs and PytorchLightning modules

Predefined folder structure. Modularity: all abstractions are split into different submodule
Rapid Experimentation. Thanks to automating pipeline with config files and hydra command line superpowers
Little Boilerplate. So pipeline can be easily modified
Main Configuration. Main config file specifies default training configuration
Experiment Configurations. Stored in a separate folder, they can be composed out of smaller configs, override chosen parameters or define everything from scratch
Experiment Tracking. Many logging frameworks can be easily integrated
Logs. All logs (checkpoints, data from loggers, chosen hparams, etc.) are stored in a convenient folder structure imposed by Hydra
Automates PyTorch Lightning training pipeline with little boilerplate, so it can be easily modified
Augmentations with albumentations described in a yaml config.
Support of timm models, pytorch-optimizer and TorchMetrics
Exponential Moving Average for a more stable training, and Stochastic Moving Average for a better generalization and just overall performance.

Project structure

The directory structure of new project looks like this:

├── src
│   ├── fulmo
│   │   ├── callbacks               <- PyTorch Lightning callbacks
│   │   ├── core                    <- PyTorch Lightning models
│   │   ├── datasets                <- PyTorch datasets
│   │   ├── losses                  <- PyTorch losses
│   │   ├── metrics                 <- PyTorch metrics  
│   │   ├── models                  <- PyTorch model architectures
│   │   ├── optimizers              <- PyTorch optimizers
│   │   ├── readers                 <- Data readers
│   │   ├── samples                 <- PyTorch samplers
│   │   ├── schedulers              <- PyTorch schedulers
│   │   └── utils
├── tests
│   ├── test_fulmo                  <- Tests
│
├── .bumpversion.cfg
├── .darglint
├── .gitignore
├── .pre-commit-config.yaml <- Configuration of hooks for automatic code formatting
├── CHANGELOG.md
├── mypy.ini
├── noxfile.py
├── poetry.lock             <- File for installing python dependencies
├── pyproject.toml          <- File for installing python dependencies
├── README.md
└── tasks.py

Workflow

Write your PyTorch model
Write your PyTorch Lightning datamodule
Write your experiment config, containing paths to your model and datamodule
Run training with chosen experiment config:

python train.py +experiment=experiment_name

Experiment Tracking

PyTorch Lightning provides built in loggers for Weights&Biases, Neptune, Comet, MLFlow, Tensorboard and CSV. To use one of them, simply add its config to configs/logger and run:

python train.py logger=logger_name

Quickstart

First, install dependencies

pip install fulmo | poetry add fulmo

Second, create your project

See examples folder.

Next, you can train model with default configuration without logging

python train.py

Or you can train model with chosen experiment config

python train.py +experiment=experiment_name

Resume from a checkpoint

# checkpoint can be either path or URL
# path should be either absolute or prefixed with `${work_dir}/`
# use quotes '' around argument or otherwise $ symbol breaks it
python train.py '+trainer.resume_from_checkpoint=${work_dir}/logs/runs/2021-06-23/16-50-49/checkpoints/last.ckpt'

TODO

Data version control
Metric learning pipeline
Integrate Cross-Batch Memory for Embedding Learning (XBM)
Image augmentation policies

Credits

This package was created with Cookiecutter and the fedejaure/cookiecutter-modern-pypackage project template.
hobogalaxy/lightning-hydra-template
Erlemar/pytorch_tempest
bonlime/pytorch-tools

jexio/fulmo