OmeZarrReader Performance Benchmarks
BrianWhitneyAI opened this issue · 11 comments
Some performance benchmarks (and followup optimization if needed) would also be awesome but maybe that can be another ticket.
Originally posted by @toloudis in #493 (comment)
I generated three different sizes of CZI images at (50 GB, 2 GB and 295 MBs) to use as benchmarks. I converted the smallest to OmeTiff and OmeZarr to benchmark the files against each other. The same process can be done with larger images in the future. These were my initial results:
| Image Format: | CZI | Ome.Tiff | Zarr |
|---|---|---|---|
| Image size | 295 Mbs | 182.39 Mbs | 243 Mbs |
| time_init | 0.01 sec. | 3.40 sec. | 0.02 sec. |
| time_delayed_array_construct | 0.06 sec. | 3.79 sec. | 0.03 sec. |
| time_random_single_chunk_read | 0.25 sec. | 4.03 sec. | 0.06 sec. |
| time_random_many_chunk_read | 0.18 sec. | 4.05 sec. | 0.04 sec. |
| time_norm_and_project | 1.53 sec. | 6.04 sec. | 0.82 sec. |
- Linux Ubuntu
- MemTotal: 16312556 kB
- Python 3.11.0
- Date: 06/27/2023
- AICSimageio 4.11.0
Wow. This is awesome.
Two things:
-
What are the shapes of the benchmark images? (256 Mbs, 2Gb, 50Gb)
-
Idk about everyone else but I would love to see this same benchmark on the 2Gb image.
The 50Gb would be awesome but definitely time consuming (at least the Ome.tiff case according to the results)
Somewhat concerning (maybe?) Is that both for CZI and OME.Zarr, the single chunk read time is longer than the many chunk read time.
To me that indicates maybe some weirdness with chunk sizing? I could definitely also be interpreting that incorrectly
The shapes of the images are as follows:
- 50 GB (161,2,40,1248,1848),
- 2 GB (5,2,40,1248,1848),
- 295 MBS (1,4,70,600,900)
Hm. For the 2Gb and 50Gb that first dim is "scenes" right?
It might be interesting to test but the only time that should really be affected is the object init time.
Scene management is done once at init. Reading shouldn't be affected at all (or maybe minimally).
Hm. For the 2Gb and 50Gb that first dim is "scenes" right?
It might be interesting to test but the only time that should really be affected is the object init time.
Scene management is done once at init. Reading shouldn't be affected at all (or maybe minimally).
These are all single scene images with dims 'TCZYX'. Right now the OmeZarrWriter writes out scenes as individual files so I had to find large single scene images to test the benchmark with.
Edit (Oh whoops i had the wrong dims above, edited now)
Oh woops. I misread the dims shape. Then ya a test of the 2Gb would be neat
Can you summarize what the individual tests in the table are doing? (For example, (1) what is a chunk in this sense? (2) And does norm_and_project happen entirely after read? is it part of a dask compute graph? )
For long time series, reading CZYX at the "middle" time value is a reasonable test that is potentially happening a lot, and might show some differences between zarr and czi. Same for "read middle z slice from fixed T but for all C".
I'm just brainstorming now but another good test would be to benchmark zarr reading with different chunk dims but maybe that's out of scope here.
I guess we also need to say what the zarr chunk size is for the tests because I imagine the perf is sensitive to it.
| Image Format: | CZI | Ome.Tiff | Zarr |
|---|---|---|---|
| Image size | 1.80 GBs | 1.14 GBs | 1.62 GBs |
| time_init | 0.02 sec. | 3.54 sec. | 0.03 sec. |
| time_delayed_array_construct | 0.07 sec. | 3.54 sec. | 0.03 sec. |
| time_random_single_chunk_read | 0.31 sec. | 4.22 sec. | 0.10 sec. |
| time_random_many_chunk_read | 0.23 sec. | 4.23 sec. | 0.06 sec. |
| time_norm_and_project | 6.90 sec. | 15.26 sec. | 2.07 sec. |
Linux Ubuntu
MemTotal: 16312556 kB
Python 3.11.0
Date: 06/27/2023
AICSimageio 4.11.0
Time_init: how long it takes to validate a file and finish the general setup.
Time_delayed_array_construct: how long it takes to construct the delayed dask array for a file.
Time_random_single_chunk_read: how long it takes to read a single chunk out of a file. I.E. "Pull just the Brightfield channel z-stack.
Time_random_many_chunk_read: Open a file, and get many chunks out of the file at once. I.E Pull the DNA and Nucleus channel z-stacks, for the middle 50% timepoints.
Time_norm_and_project: how long a norm and project through Z takes
under various chunk dims configurations. Builds a dask array stack of these projections and computes to array.
We love Zarr. cc @joshmoore
One more quick question for you @BrianWhitneyAI, we know the image shape but what is the chunk shape for each image? I assume chunked by ZYX?
We love Zarr. cc @joshmoore
One more quick question for you @BrianWhitneyAI, we know the image shape but what is the chunk shape for each image? I assume chunked by ZYX?
yep!