Compress Seg

Library for compressing and decompressing image segmentation (adapted from neuroglancer)

import compressed_segmentation as cseg

sx, sy, sz = (128,128,128)
dtype = np.uint64
order = 'C'

labels = np.arange(0, sx*sy*sz, dtype=dtype).reshape((sx,sy,sz), order=order)
compressed = cseg.compress(labels, order=order)
recovered = cseg.decompress(
    compressed, (sx,sy,sz) dtype=dtype, order=order
)

arr = CompressedSegmentationArray(
    compressed, shape=(sx,sy,sz), dtype=dtype
)
label = arr[54,32,103] # random access to single voxels w/o decompressing
uniq_labels = arr.labels() # get all distinct values w/o decompressing
binary2 = arr.remap({ 1: 2 }, preserve_missing_labels=False) # remap labels in segmentation w/o decompressing
recovered = arr.numpy() # decompress to a numpy array, same as decompress
124213 in arr # test if a value is in the array

cseg compress connectomics.npy
cseg decompress connectomics.npy.cseg --volume-size 512,512,512 --bytes 4

NOTE: This repository is the PyPI distribution repo but is based on work done by Jeremy Maitin-Shepard (Google), Stephen Plaza (Janelia Research Campus), and William Silversmith (Princeton) here: https://github.com/janelia-flyem/compressedseg

This library contains routined to decompress and compress segmentation and to manipulate compressed segmentation data defined by the neuroglancer project. compressed_segmentation essentially renumbers large bit width labels to smaller ones in chunks. This provides for large reductions in memory usage and higher compression.

Note that limitations in the compressed_segmentation format restrict the size of the chunk that can be compressed. As this limitation is data dependent, for example a random array with 1024 labels passes testing at 256x256x128, but 256x256x256 often does not.

Features

Compression and decompression
Random access to voxels without decompression
Read out unique values without decompression
Remap labels without decompression
Command line interface for numpy files
(TBD) Interface to relabel and manipulate segmentation from the compressed data
C++, Python, and Go interface (see original repo for Golang)

C++ Compilation

Compiling as a shared library. Feel free to subsititute e.g. clang for the C++ compiler.

g++ -std=c++11 -O3 -fPIC -shared -I./include src/compress_segmentation.cc src/decompress_segmentation.cc -o compress_segmentation.so

Python Installation

`pip` Binary Installation

$ pip install compressed-segmentation

$ python
>>> import compressed_segmentation as cseg
>>> help(cseg)

If there are pre-built binaries available for your architecture this should just work.

`pip` Source Installation

If you need to build from source, you will need to have a C++ compiler installed:

$ sudo apt-get install g++ python3-dev 
$ pip install numpy
$ pip install compressed-segmentation

$ python
>>> import compressed_segmentation as cseg
>>> help(cseg)

Direct Installation

Requires a C++ compiler such as g++ or clang.

Works with both Python 2 and 3. Encodes from / decodes to 3D or 4D numpy ndarrays.

$ sudo apt-get install g++ python3-dev 
$ pip install -r requirements.txt
$ python setup.py install

$ python
>>> import compressed_segmentation as cseg
>>> help(cseg)

License

Please see the licenses in this repo.

seung-lab/compressedseg