This repository contains the source code for the MorphoCluster application described in [Schroeder2020].
The web application runs in a Docker container.
Clone the morphocluster repository and checkout the current stable branch:
git clone -b 0.2.x https://github.com/morphocluster/morphocluster.git
Decide where to store the data, e.g.
/data/morphocluster
, and create the required directories, e.g.mkdir -p /data/morphocluster/{postgres,data}
. Make a copy ofdocker-compose.default.yml
in the same directory, name itdocker-compose.yml
and edit the relevant parts (e.g. paths). Make a copy ofenvironment.default.yml
in the same directory, name itenvironment.yml
and edit the relevant parts (e.g. for CUDA).Open a shell in the repository folder and build environment:
docker-compose up --build
This will take a long while when run for the first time.In a new terminal, connect to the MorphoCluster container and set it up:
$ docker-compose exec morphocluster bash # Activate conda environment (base) root@abc123...:/code# . ./activate (morphocluster) root@abc123...:/code# # Create a user account flask add-user test-user Adding user test-user: Password: <hidden> Retype Password: <hidden>
Data preparation. MorphoCluster accepts input data in ZIP files containing the image files and an
index.csv
file with two columns:object_id
(a unique identifier for every object) andpath
(image file path inside the archive). EcoTaxa export files have to be converted (see below).Place the image archive and the deep learning model parameters into the data directory that is mounted inside the container.
# Connect to MorphoCluster container and set up the environment (if not done before) $ docker-compose exec morphocluster bash (base) root@abc123...:/code# . ./activate (morphocluster) root@abc123...:/code# # Change into data directory $ cd /data # 0. Optionally convert EcoTaxa archive: $ morphocluster fix-ecotaxa archive.zip # 1. Calculate deep learning image features. # Using the GPU (see below) will considerably speed this up. # If --parameters-fn is not supplied, ImageNet-trained model weights will be used. # --input-mean and --input-std should then be supplied with the estimated mean color values. $ morphocluster features [--parameters-fn model_state.pth] [--input-mean 0.9,0.9,0.9] [--input-std 1,1,1] archive.zip features.h5 # 2a. Cluster the features. $ morphocluster cluster [--pca 64] --min-cluster-size 128 features.h5 tree-128.zip # 2b. When repeating the clustering in the next iteration of the MorphoCluster process, supply the previously exported tree and reduce the cluster size. $ morphocluster cluster [--pca 64] --min-cluster-size 64 features.h5 tree-64.zip --tree /data/export/2020-05-15-10-34-34--3--tree-128.zip # 3. Import objects into the web application flask load-objects archive.zip # 4. Import features into the web application flask load-features [--pca 64] features.h5 # 5. Import project into the web application flask load-project tree.zip
Keep in mind that the paths are inside the Docker container, not on the host.
Connect to the MorphoCluster service using the browser (http://localhost:8000/). Validate and Grow the found clusters and save the result. It will be placed in
/data/export/
in the container.Repeat clustering, tree import, validation and growing.
- Docker (Version 20 or higher)
- Docker Compose (Version 1.25 or higher)
- The above steps were only tested on Linux.
By default, MorphoCluster uses the CPU to calculate deep learning image features.
It is strongly advisable to use the GPU for that to speed up feature extraction dramatically.
To enable feature extraction on the GPU, modify environment.yml
to exclude cpuonly
and include the correct cudatoolkit
version.
This has to be done before running docker-compose up --build
.
Read about the requirements for running CUDA containers.
Currently, docker-compose
does not directly support NVIDIA docker (see #1073, #6691).
It is therefore advisable to run the feature extraction direcly on the host.
For users without the privilege to execute docker commands on the host system (e.g. for security reasons), it is possible to connect to the docker container directly via SSH.
Make sure that port 22 is exposed in
docker-compose.yml
.Create a public key
# Generate key pair (if not done already): $ ssh-keygen -t rsa -b 4096 # Show public key. Copy this to your clipboard. $ cat ~/.ssh/id_rsa.pub # It looks a bit like this: ssh-rsa asdxyz ...
Create
docker/morphocluster/authorized_keys
from thedocker/morphocluster/authorized_keys.default
template and paste the public key created in the previous step.After starting the morphocluster docker service, you can now connect to the docker container directly:
$ ssh root@host -p 8022 -i ~/.ssh/id_rsa -o PubkeyAuthentication=yes # Inside the container, change into /code and activate the environment $ cd /code $ . activate
[Schroeder2020] | Schröder, S., Kiko, R., & Koch, R. (2020). "MorphoCluster: Efficient Annotation of Plankton images by Clustering" arXiv:2005.01595. |