Tools for segmentation and tracking in microscopy build on top of SegmentAnything. Segment and track objects in microscopy images interactively with a few clicks!
We implement napari applications for:
- interactive 2d segmentation
- interactive 3d segmentation
- interactive tracking of 2d image data
Early beta version
This is an early beta version. Any feedback is welcome, but please be aware that the functionality is under active development and that several features are not finalized or thoroughly tested yet. Once the functionality has matured we plan to release the interactive annotation applications as napari plugins.
If you run into any problems or have questions please open an issue or reach out via image.sc using the tag micro-sam
and tagging @constantinpape.
New in version 0.02
- We have added support for bounding box prompts (see the gif below), which provide better segmentation results than points in many cases.
- Interactive tracking now uses a better heuristic to propagate masks across time, leading to better automatic tracking results.
- And have fixed several small bugs.
We implement applications for fast interactive 2d and 3d segmentation as well as tracking.
- Left: interactive 2d segmentation
- Middle: interactive 3d segmentation
- Right: interactive tracking
We require these dependencies:
We recommend to use conda and provide two environment files with all necessary requirements:
environment_gpu.yaml
: sets up an environment with GPU support.environment_cpu.yaml
: sets up an environment with CPU support.
To install via conda first clone this repository:
git clone https://github.com/computational-cell-analytics/micro-sam
and
cd micro_sam
Then create either the GPU or CPU environment via
conda env create -f <ENV_FILE>.yaml
Then activate the environment via
conda activate sam
And install our napari applications and the micro_sam
library via
pip install -e .
Troubleshooting:
- On some systems
conda
is extremely slow and cannot resolve the environment in the stepconda env create ...
. You can usemamba
instead, which is a faster re-implementation ofconda
. It can resolve the environment in less than a minute on any system we tried. Check out this link for how to installmamba
. Once you have installed it, runmamba env create -f <ENV_FILE>.yaml
to create the env. - Installation on MAC with a M1 or M2 processor:
- The pytorch installation from
environment_cpu.yaml
does not work with a MAC that has an M1 or M2 processor. Instead you need to:- Create a clean environment:
conda create -c conda-forge python pip -n sam
- Activate it va
conda activate sam
- Follow the instructions for how to install pytorch for MAC via conda from pytorch.org.
- Install additional dependencies:
conda install -c conda-forge napari python-elf tqdm
- Install SegmentAnything:
pip install git+https://github.com/facebookresearch/segment-anything.git
- Install
micro_sam
by runningpip install -e .
in this folder.
- Create a clean environment:
- The pytorch installation from
After installing the micro_sam
python application the three interactive annotation tools can be started from the command line or from a python script (see details below).
They are built with napari to implement the viewer and user interaction. If you are not familiar with napari yet, start here.
To use the apps the functionality of napari point layers, napari shape layers and napari labels layers is of particular importance.
Note: the screenshots and tutorials do not show how to use bounding boxes for prompts yet. You can use the box_prompts
layer for them in all three tools, and they can be used as a replacement or in combination with the point prompts.
The application for 2d segmentation can be started in two ways:
- Via the command line with the command
micro_sam.annotator_2d
. Runmicro_sam.annotator_2d -h
for details. - From a python script with the function
micro_sam.sam_annotator.annotator_2d
. Check out examples/sam_annotator_2d for details.
Below you can see the interface of the application for a cell segmentation example:
The most important parts of the user interface are:
- The napari layers that contain the image, segmentations and prompts:
prompts
: point layer that is used to provide prompts to SegmentAnything. Positive prompts (green points) for marking the object you want to segment, negative prompts (red points) for marking the outside of the object.current_object
: label layer that contains the object you're currently segmenting.committed_objects
: label layer with the objects that have already been segmented.auto_segmentation
: label layer results from using SegmentAnything for automatic instance segmentation.raw
: image layer that shows the image data.
- The prompt menu for changing the currently selected point from positive to negative and vice versa. This can also be done by pressing
t
. - The menu for automatic segmentation. Pressing
Segment All Objects
will run automatic segmentation (this can take few minutes if you are using a CPU). The results will be displayed in theauto_segmentation
layer. - The menu for interactive segmentation. Pressing
Segment Object
(ors
) will run segmentation for the current prompts. The result is displayed incurrent_object
- The menu for commiting the segmentation. When pressing
Commit
(orc
) the result from the selected layer (eithercurrent_object
orauto_segmentation
) will be transferred from the respective layer tocommitted_objects
.
Check out this video for an overview of the interactive 2d segmentation functionality.
The application for 3d segmentation can be started as follows:
- Via the command line with the command
micro_sam.annotator_3d
. Runmicro_sam.annotator_3d -h
for details. - From a python script with the function
micro_sam.sam_annotator.annotator_3d
. Check out examples/sam_annotator_3d for details.
The most important parts of the user interface are listed below. Most of these elements are the same as for the 2d segmentation app.
- The napari layers that contain the image, segmentation and prompts. Same as for the 2d segmentation app but without the
auto_segmentation
layer. - The prompt menu.
- The menu for interactive segmentation.
- The 3d segmentation menu. Pressing
Segment Volume
(orv
) will extend the segmentation for the current object across the volume. - The menu for committing the segmentation.
Check out this video for an overview of the interactive 3d segmentation functionality.
The application for interactive tracking (of 2d data) can be started as follows:
- Via the command line with the command
micro_sam.annotator_tracking
. Runmicro_sam.annotator_tracking -h
for details. - From a python script with the function
micro_sam.sam_annotator.annotator_tracking
. Check out examples/sam_annotator_tracking for details.
The most important parts of the user interface are listed below. Most of these elements are the same as for the 2d segmentation app.
- The napari layers thaat contain the image, segmentation and prompts. Same as for the 2d segmentation app but without the
auto_segmentation
layer,current_tracks
andcommitted_tracks
are the equivalent ofcurrent_object
andcommitted_objects
. - The prompt menu.
- The menu with tracking settings:
track_state
is used to indicate that the object you are tracking is dividing in the current frame.track_id
is used to select which of the tracks after divsion you are following. - The menu for interactive segmentation.
- The tracking menu. Press
Track Object
(orv
) to track the current object across time. - The menu for committing the current tracking result.
Check out this video for an overview of the interactive tracking functionality.
- By default, the applications pre-compute the image embeddings produced by SegmentAnything and store them on disc. If you are using a CPU this step can take a while for 3d data or timeseries (you will see a progress bar with a time estimate). If you have access to a GPU without graphical interface (e.g. via a local computer cluster or a cloud provider), you can also pre-compute the embeddings there and then copy them to your laptop / local machine to speed this up. You can use the command
micro_sam.precompute_embeddings
for this (it is installed with the rest of the applications). You can specify the location of the precomputed embeddings via theembedding_path
argument. - Most other processing steps are very fast even on a CPU, so interactive annotation is possible. An exception is the automatic segmentation step (2d segmentation), which takes several minutes without a GPU (depending on the image size). For large volumes and timeseries segmenting an object in 3d / tracking across time can take a couple settings with a CPU (it is very fast with a GPU).
- You can also try using a smaller version of the SegmentAnything model to speed up the computations. For this you can pass the
model_type
argument and either set it tovit_l
orvit_b
(default isvit_h
). However, this may lead to worse results. - You can save and load the results from the
committed_objects
/committed_tracks
layer to correct segmentations you obtained from another tool (e.g. CellPose) or to save intermediate annotation results. The results can be saved viaFile->Save Selected Layer(s) ...
in the napari menu (see the tutorial videos for details). They can be loaded again by specifying the corresponding location via thesegmentation_result
(2d and 3d segmentation) ortracking_result
(tracking) argument.
- SegmentAnything does not work well for very small or fine-graind objects (e.g. filaments).
- For the automatic segmentation functionality we currently rely on the automatic mask generation provided by SegmentAnything. It is slow and often misses objects in microscopy images. For now, we only offer this functionality in the 2d segmentation app; we are working on improving it and extending it to 3d segmentation and tracking.
- Prompt bounding boxes do not provide the full functionality for tracking yet (they cannot be used for divisions or for starting new tracks). See also computational-cell-analytics#23.
After installation the micro_sam
python library is available, which provides several utility functions for using SegmentAnything with napari. Check out examples/image_series_annotator.py for an example application for segmenting objects in an image series built with it.
If you are using this repository in your research please cite
- SegmentAnything
- and our repository on zenodo (we are working on a publication)
There are two other napari plugins build around segment anything:
- https://github.com/MIC-DKFZ/napari-sam (2d and 3d support)
- https://github.com/JoOkuma/napari-segment-anything (only 2d support)
Compared to these we support more applications (2d, 3d and tracking), and aim to further extend and specialize SegmentAnything for microscopy data. WebKnossos also offers integration of SegmentAnything for interactive segmentation.