CVPR2023 Highlight | MAESTER: Masked Autoencoder Guided Segmentation at Pixel Resolution for Accurate, Self-Supervised Subcellular Structure Recognition

Check out the Paper! and our Youtube Talk!

💥 Introduction

We introduce MAESTER (Masked AutoEncoder guided SegmenTation at pixEl Resolution), a self-supervised method for accurate, subcellular structure segmentation at pixel resolution. MAESTER treats volume electron microscopy(vEM) image segmentation as a representation learning and clustering problem. Specifically, MAESTER learns semantically meaningful token representations of multi-pixel image patches while simultaneously maintaining a sufficiently large field of view for contextual learning. We also develop a cover-and-stride inference strategy to achieve pixel-level subcellular strueture segmentation.

⚙️ Installation

Clone the repository:

git clone https://github.com/bowang-lab/MAESTER

Set up the environment:

poetry install
poetry shell
pip install torch==2.0.1 torchvision==0.15.2

Download the trained model for demo
- Google drive: https://drive.google.com/drive/folders/143W_VSl5ONE3NGbnI0i19S8lBRml7lRz?usp=sharing
- Put it under ./MAESTER/model_weights/
Dataset:
- Download the betaSeg dataset by:
```
wget https://cloud.mpi-cbg.de/index.php/s/UJopHTRuh6f4wR8/download
```
- and unzip the dataset, put it under ./MAESTER/data/

🎉 Example

Check out our detailed demo:
- Inference with MAESTER ./examples/inference_demo.ipynb.

📝 To-do

Add inference demo
Add scalable inference example
Add DDP training example

📄 Citation

@InProceedings{Xie_2023_CVPR,
    author    = {Xie, Ronald and Pang, Kuan and Bader, Gary D. and Wang, Bo},
    title     = {MAESTER: Masked Autoencoder Guided Segmentation at Pixel Resolution for Accurate, Self-Supervised Subcellular Structure Recognition},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {3292-3301}
}

Acknowledgement

This repository is built upon MAE.

JonasHell/MAESTER