OME-Zarr Conversion Pipeline

Nextflow pipeline which converts Bioformats-compatible images to NGFF (e.g. OME-Zarr) format using bioformats2raw.

Why not just use bioformats2raw directly? This pipeline encapsulates all the dependencies, so you don't need to install Java, blosc, etc. It also lets you distribute work on any HPC cluster.

Quick Start

The only software requirements for running this pipeline are Nextflow (version 20.10.0 or greater) and either Docker or Singularity (version 3.5 or greater). If you are running on an HPC cluster, ask your system administrator to install Singularity on all the cluster nodes.

To install Nextflow:

curl -s https://get.nextflow.io | bash

Alternatively, you can install it as a conda package:

conda create --name nextflow -c bioconda nextflow

To install Singularity on CentOS Linux:

sudo yum install singularity

Now you can run the pipeline and it will download everything else it needs. First, prepare a samplesheet with your input data that looks as follows:

samplesheet.csv:

id,image,output_path
image1,/path/to/set1/image1.czi,set1
image2,/path/to/set1/image2.czi,set1
image3,/path/to/set2/image1.czi,set2

Each row represents one input image in any Bioformats-compatible format (Zeiss CZI in the example above). The output_path is a folder relative to the --outdir parameter and can be left empty to place output Zarrs directly in the output_path.

The following command will convert the images listed in samplesheet.csv to OME-Zarrs in the ./output directory, using 40 cores for each conversion process, and zlib compression.

    nextflow run JaneliaSciComp/nf-omezarr -profile singularity \
        --input samplesheet.csv --outdir ./output --compression zlib --cpus 40

By default, the Zarr chunk size is set to 128,128,128. You can customize the chunk size of the zarr using the --chunk_size parameter, e.g.

    nextflow run JaneliaSciComp/nf-omezarr -profile singularity \
        --input samplesheet.csv --outdir ./output --chunk_size 1920,1920,1

This pipeline is nf-core-compatible and reuses pipeline infrastructure from the nf-core project, including the ability to use nf-core institutional profiles that let you run on many university clusters without additional configuration. For example, to run this pipeline on the Janelia cluster:

    nextflow run JaneliaSciComp/nf-omezarr -profile janelia \
         --input samplesheet.csv --outdir ./output

Pipeline options

Define the input data and bioformats2raw parameters.

Parameter	Description	Type	Required
`input`	Path to comma-separated file containing information about the samples in the experiment. Help You will need to create a design file with information about the samples in your experiment before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 4 columns, and a header row. See usage docs.	`string`	True
`outdir`	The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure.	`string`	True
`bioformats2raw_opts`	Extra options for Bioformats2raw	`string`
`chunk_size`	Chunk size for Zarr in X,Y,Z order. Default: 128,128,128	`string`
`compression`	How the blocks will be compressed. The default "blosc" compression uses LZ4. The "zlib" compression is slower but usually achieves a higher compression ratio. More information is available in the [https://jzarr.readthedocs.io/en/latest/tutorial.html#compressors](JZarr documentation). Default: blosc	`string`
`cpus`	Number of cores to allocate for bioformats2raw. Default: 10	`integer`
`memory`	Amount of memory to allocate for bioformats2raw. Default: 36.G	`string`

Institutional config options

Parameters used to describe centralised config profiles. These should not be edited.

Parameter	Description	Type	Default	Hidden
`custom_config_version`	Git commit id for Institutional configs.	`string`	master	True
`custom_config_base`	Base directory for Institutional configs. Help If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.	`string`	https://raw.githubusercontent.com/nf-core/configs/master	True
`config_profile_name`	Institutional config name.	`string`		True
`config_profile_description`	Institutional config description.	`string`		True
`config_profile_contact`	Institutional config contact information.	`string`		True
`config_profile_url`	Institutional config URL link.	`string`		True

Max job request options

Set the top limit for requested resources for any single job.

Parameter	Description	Type	Default	Hidden
`max_cpus`	Maximum number of CPUs that can be requested for any single job. Help Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. `--max_cpus 1`	`integer`	16	True
`max_memory`	Maximum amount of memory that can be requested for any single job. Help Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. `--max_memory '8.GB'`	`string`	128.GB	True
`max_time`	Maximum amount of time that can be requested for any single job. Help Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`	`string`	240.h	True

Generic options

Less common options for the pipeline, typically set in a config file.

Parameter	Description	Type	Default	Hidden
`help`	Display help text.	`boolean`		True
`version`	Display version and exit.	`boolean`		True
`email`	Email address for completion summary. Help Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file (`~/.nextflow/config`) then you don't need to specify this on the command line for every run.	`string`
`email_on_fail`	Email address for completion summary, only when pipeline fails. Help An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully.	`string`		True
`plaintext_email`	Send plain-text email instead of HTML.	`boolean`		True
`monochrome_logs`	Do not use coloured log outputs.	`boolean`		True
`hook_url`	Incoming hook URL for messaging service Help Incoming hook URL for messaging service. Currently, MS Teams and Slack are supported.	`string`		True
`validate_params`	Boolean whether to validate parameters against the schema at runtime	`boolean`	True	True
`validationShowHiddenParams`	Show all params when using `--help` Help By default, parameters set as hidden in the schema are not shown on the command line when a user runs with `--help`. Specifying this option will tell the pipeline to show all parameters.	`boolean`		True
`validationFailUnrecognisedParams`	Validation of parameters fails when an unrecognised parameter is found. Help By default, when an unrecognised parameter is found, it returns a warinig.	`boolean`		True
`validationLenientMode`	Validation of parameters in lenient more. Help Allows string values that are parseable as numbers or booleans. For further information see JSONSchema docs.	`boolean`		True