/medigan

medigan - A Python Library of Pretrained Generative Models for Medical Image Synthesis

Primary LanguagePythonMIT LicenseMIT

medigan

License Continuous integration PyPI version Conda version DOI arXiv

medigan stands for medical generative (adversarial) networks. medigan provides user-friendly medical image synthesis and allows users to choose from a range of pretrained generative models to generate synthetic datasets. These synthetic datasets can be used to train or adapt AI models that perform clinical tasks such as lesion classification, segmentation or detection.

See below how medigan can be run from the command line to generate synthetic medical images.

medigan can be run directly from the command line to generate synthetic medical images

Features:

  • ❌ Problem 1: Data scarcity in medical imaging.

  • ❌ Problem 2: Scarcity of readily reusable generative models in medical imaging.

  • βœ… Solution: medigan

    1. dataset sharing via generative models 🎁
    2. data augmentation 🎁
    3. domain adaptation 🎁
    4. synthetic data evaluation method testing with multi-model datasets 🎁

Instead of training your own, use one of the generative models from medigan to generate synthetic data.

Search and find a model in medigan using search terms (e.g. "Mammography" or "Endoscopy").

Contribute your own generative model to medigan to increase its visibility, re-use, and impact.

Available models

Output type Modality Model type Output size Base dataset Output examples model_id Hosted on Reference
Breast Calcification mammography dcgan 128x128 Inbreast sample 00001_DCGAN_MMG_CALC_ROI Zenodo (5187714)
Breast Mass mammography dcgan 128x128 Optimam sample 00002_DCGAN_MMG_MASS_ROI Zenodo (5188557) Alyafi et al (2019)
Breast Density Transfer mammography cyclegan 1332x800 BCDR sample 00003_CYCLEGAN_MMG_DENSITY_FULL Zenodo (5547263) Garrucho et al (2022)
Breast Mass with Mask mammography pix2pix 256x256 BCDR sample
sample
00004_PIX2PIX_MMG_MASSES_W_MASKS Zenodo (7093759)
Breast Mass mammography dcgan 128x128 BCDR sample 00005_DCGAN_MMG_MASS_ROI Zenodo (6555188) Szafranowska et al (2022)
Breast Mass mammography wgan-gp 128x128 BCDR sample 00006_WGANGP_MMG_MASS_ROI Zenodo (6554713) Szafranowska et al (2022)
Brain Tumors on Flair, T1, T1c, T2 with Masks brain MRI inpaint GAN 256x256 BRATS 2018 sample
sample
sample
sample
sample
sample
00007_INPAINT_BRAIN_MRI Zenodo (7041737) Kim et al (2020)
Breast Mass (Mal/Benign) mammography c-dcgan 128x128 CBIS-DDSM sample 00008_C-DCGAN_MMG_MASSES Zenodo (6647349) Osuala et al (2024)
Polyp with Mask endoscopy pggan 256x256 HyperKvasir sample
sample
00009_PGGAN_POLYP_PATCHES_W_MASKS Zenodo (6653743) Thambawita et al (2022)
Polyp with Mask endoscopy fastgan 256x256 HyperKvasir sample
sample
00010_FASTGAN_POLYP_PATCHES_W_MASKS Zenodo (6660711) Thambawita et al (2022)
Polyp with Mask endoscopy singan β‰ˆ250x250 HyperKvasir sample
sample
00011_SINGAN_POLYP_PATCHES_W_MASKS Zenodo (6667944) Thambawita et al (2022)
Breast Mass (Mal/Benign) mammography c-dcgan 128x128 BCDR sample 00012_C-DCGAN_MMG_MASSES Zenodo (6755693)
Breast Density Transfer MLO mammography cyclegan 1332x800 OPTIMAM sample 00013_CYCLEGAN_MMG_DENSITY_OPTIMAM_MLO Zenodo (6818095) Garrucho et al (2022)
Breast Density Transfer CC mammography cyclegan 1332x800 OPTIMAM sample 00014_CYCLEGAN_MMG_DENSITY_OPTIMAM_CC Zenodo (6818103) Garrucho et al (2022)
Breast Density Transfer MLO mammography cyclegan 1332x800 CSAW sample 00015_CYCLEGAN_MMG_DENSITY_CSAW_MLO Zenodo (6818105) Garrucho et al (2022)
Breast Density Transfer CC mammography cyclegan 1332x800 CSAW sample 00016_CYCLEGAN_MMG_DENSITY_CSAW_CC Zenodo (6818107) Garrucho et al (2022)
Lung Nodules chest x-ray dcgan 128x128 NODE21 sample 00017_DCGAN_XRAY_LUNG_NODULES Zenodo (6943691)
Lung Nodules chest x-ray wgan-gp 128x128 NODE21 sample 00018_WGANGP_XRAY_LUNG_NODULES Zenodo (6943761)
Chest Xray Images chest x-ray pggan 1024x1024 ChestX-ray14 sample 00019_PGGAN_CHEST_XRAY Zenodo (6943803)
Chest Xray Images chest x-ray pggan 1024x1024 ChestX-ray14 sample 00020_PGGAN_CHEST_XRAY Zenodo (7046280) Segal et al (2021)
Brain T1-T2 MRI Modality Transfer brain MRI cyclegan 224x192 CrossMoDA 2021 sample 00021_CYCLEGAN_BRAIN_MRI_T1_T2 Zenodo (7074555) Joshi et al (2022)
Cardiac MRI Age Transfer cardiac MRI wgan 256x256 UK Biobank sample 00022_WGAN_CARDIAC_AGING Zenodo (7446930) Campello et al (2022)
Breast DCE-MRI Contrast Injection breast DCE-MRI pix2pixHD 512x512 Duke Dataset sample 00023_PIX2PIXHD_BREAST_DCEMRI Zenodo (10210944) Osuala et al (2023)

Model information can be found in:

  • model documentation (e.g. the parameters of the models' generate functions)
  • global.json file (e.g. metadata for model description, selection, and execution)
  • medigan paper (e.g. analysis and comparisons of models and FID scores)

Installation

To install the current release, simply run:

pip install medigan

Or, alternatively via conda:

conda install -c conda-forge medigan

Getting Started

Examples and notebooks are located at examples folder

Documentation is available at medigan.readthedocs.io

Generation example

DCGAN

Create mammography masses with labels (malignant or benign) using a class-conditional DCGAN model.

# import medigan and initialize Generators
from medigan import Generators
generators = Generators()

# generate 8 samples with model 8 (00008_C-DCGAN_MMG_MASSES). 
# Also, auto-install required model dependencies.
generators.generate(model_id=8, num_samples=8, install_dependencies=True)

sample

The synthetic images in the top row show malignant masses (breast cancer) while the images in the bottom row show benign masses. Given such images with class information, image classification models can be (pre-)trained.

CYCLEGAN

Create mammograms translated from Low-to-High Breast Density using CYCLEGAN model

from medigan import Generators
generators = Generators()
# model 3 is "00003_CYCLEGAN_MMG_DENSITY_FULL"
generators.generate(model_id=3, num_samples=1)

sample β†’ sample

Search Example

Search for a model inside medigan using keywords

# import medigan and initialize Generators
from medigan import Generators
generators = Generators()

# list all models
print(generators.list_models())

# search for models that have specific keywords in their config
keywords = ['DCGAN', 'Mammography', 'BCDR']
results = generators.find_matching_models_by_values(keywords)

Get Model as Dataloader

We can directly receive a torch.utils.data.DataLoader object for any of medigan's generative models.

from medigan import Generators
generators = Generators()
# model 4 is "00004_PIX2PIX_MMG_MASSES_W_MASKS"
dataloader = generators.get_as_torch_dataloader(model_id=4, num_samples=3)

Visualize the contents of the dataloader.

from matplotlib import pyplot as plt
import numpy as np

plt.figure()
# subplot with 2 rows and len(dataloader) columns
f, img_array = plt.subplots(2, len(dataloader)) 

for batch_idx, data_dict in enumerate(dataloader):
    sample = np.squeeze(data_dict.get("sample"))
    mask = np.squeeze(data_dict.get("mask"))
    img_array[0][batch_idx].imshow(sample, interpolation='nearest', cmap='gray')
    img_array[1][batch_idx].imshow(mask, interpolation='nearest', cmap='gray')
plt.show()

sample

Visualize A Model

With our interface, it is possible to generate sample by manually setting the conditional inputs or latent vector values. The sample is updated in realtime, so it's possible to observe how the images changes when the parameters are modified. The visualization is available only for models with accessible input latent vector. Depending on a model, a conditional input may be also available or synthetic segmentation mask.

from medigan import Generators

generators = Generators()
# model 10 is "00010_FASTGAN_POLYP_PATCHES_W_MASKS"
generators.visualize(10)

sample

Contribute A Model

Create an init.py file in your model's root folder.

Next, run the following code to contribute your model to medigan.

  • Your model will be stored on Zenodo.

  • Also, a Github issue will be created to add your model's metadata to medigan's global.json.

  • To do so, please provide a github access token (get one here) and a zenodo access token (get one here), as shown below. After creation, the zenodo access token may take a few minutes before being recognized in zenodo API calls.

from medigan import Generators
generators = Generators()

# Contribute your model
generators.contribute(
    model_id = "00100_YOUR_MODEL", # assign an ID
    init_py_path ="path/ending/with/__init__.py",
    model_weights_name = "10000",
    model_weights_extension = ".pt",
    generate_method_name = "generate", # in __init__.py
    dependencies = ["numpy", "torch"], 
    creator_name = "YOUR_NAME",
    creator_affiliation = "YOUR_AFFILIATION",
    zenodo_access_token = 'ZENODO_ACCESS_TOKEN',
    github_access_token = 'GITHUB_ACCESS_TOKEN',

Thank you for your contribution!

You will soon receive a reply in the Github issue that you created for your model by running generators.contribute().

Contributions in General

We welcome contributions to medigan. Please send us an email or read the contributing guidelines regarding contributing to the medigan project.

Reference

If you use a medigan model in your work, please cite its respective publication (see references).

Please also consider citing the medigan paper:

Osuala, R., Skorupko, G., Lazrak, N., Garrucho, L., GarcΓ­a, E., Joshi, S., ... & Lekadir, K. (2023). medigan: a Python library of pretrained generative models for medical image synthesis. Journal of Medical Imaging, 10(6), 061403.

BibTeX entry:

@article{osuala2023medigan,
  title={medigan: a Python library of pretrained generative models for medical image synthesis},
  author={Osuala, Richard and Skorupko, Grzegorz and Lazrak, Noussair and Garrucho, Lidia and Garc{\'\i}a, Eloy and Joshi, Smriti and Jouide, Socayna and Rutherford, Michael and Prior, Fred and Kushibar, Kaisar and others},
  journal={Journal of Medical Imaging},
  volume={10},
  number={6},
  pages={061403},
  year={2023},
  publisher={SPIE}
}