High-content screening ngff metadata schema empty image
mfranzon opened this issue · 4 comments
I am trying to open a .zarr folder with this metadata schema choosing napari-plugin as reader, napari does not raise any error but the image loaded is just an "empty" array.
This is my folder structure:
As you see, I have just one dataset path (0)
This is the result after I drag and drop it on napari with the plugin:
This is the output of the napari with -vvv :
`10:22:39 DEBUG Created nested FSStore(/test2.zarr, r, {'dimension_separator': '/', 'normalize_keys': False})
10:22:39 DEBUG 0.4 matches None?
10:22:39 DEBUG 0.3 matches None?
10:22:39 DEBUG 0.2 matches None?
10:22:39 DEBUG V01:None v. 0.1
10:22:39 DEBUG treating /test2.zarr [zgroup] as Plate
10:22:39 INFO root_attr: plate
10:22:39 DEBUG {'acquisitions': [{'id': 1, 'name': 'single acquisition'}], 'columns': [{'name': '1'}], 'field_count': 1, 'name': 'test', 'rows': [{'name': 'A'}], 'version': '0.3', 'wells': [{'path': 'A/1'}]}
10:22:39 INFO plate_data: {'acquisitions': [{'id': 1, 'name': 'single acquisition'}], 'columns': [{'name': '1'}], 'field_count': 1, 'name': 'test', 'rows': [{'name': 'A'}], 'version': '0.3', 'wells': [{'path': 'A/1'}]}
10:22:39 DEBUG open(ZarrLocation(/test2.zarr/A/1))
10:22:39 DEBUG Created nested FSStore(/test2.zarr/A/1, r, {'dimension_separator': '/', 'normalize_keys': False})
10:22:39 DEBUG 0.4 matches None?
10:22:39 DEBUG 0.3 matches None?
10:22:39 DEBUG 0.2 matches None?
10:22:39 DEBUG V01:None v. 0.1
10:22:39 DEBUG treating /test2.zarr/A/1 [zgroup] as Well
10:22:39 INFO root_attr: well
10:22:39 DEBUG {'images': [{'path': '0'}], 'version': '0.3'}
10:22:39 INFO well_data: {'images': [{'path': '0'}], 'version': '0.3'}
10:22:39 DEBUG open(ZarrLocation(/test2.zarr/A/1/0))
10:22:39 DEBUG Created nested FSStore(/test2.zarr/A/1/0, r, {'dimension_separator': '/', 'normalize_keys': False})
10:22:39 DEBUG 0.4 matches 0.3?
10:22:39 DEBUG 0.3 matches 0.3?
10:22:39 WARNING version mismatch: detected:FormatV03, requested:FormatV04
10:22:39 DEBUG Created nested FSStore(/test2.zarr/A/1/0, r, {'dimension_separator': '/', 'normalize_keys': False})
10:22:39 DEBUG treating /test2.zarr/A/1/0 [zgroup] as Multiscales
10:22:39 INFO root_attr: multiscales
10:22:39 DEBUG [{'version': '0.3', 'axes': [{'name': 'y', 'type': 'space', 'unit': 'centimeter'}, {'name': 'x', 'type': 'space', 'unit':
'centimeter'}], 'datasets': [{'path': '0'}], 'metadata': {'kwargs': {'axes_names': ['y', 'x']}}}]
10:22:39 INFO root_attr: omero
10:22:39 DEBUG {'id': 1, 'version': '0.3'}
10:22:39 INFO root_attr: channels
10:22:39 DEBUG [{'active': True, 'coefficient': 1, 'color': '0000FF', 'family': 'linear', 'inverted': False, 'label': 'LaminB1', 'window': {'end': 1500, 'max': 65535, 'min': 0, 'start': 0}}]
10:22:39 INFO root_attr: rdefs
10:22:39 DEBUG {'defaultT': 0, 'defaultZ': 118, 'model': 'color'}
10:22:39 INFO datasets [{'path': '0'}]
10:22:39 INFO resolution: 0
10:22:39 INFO - shape ('y', 'x') = (8640, 7680)
10:22:39 INFO - chunks = ['2160', '2560']
10:22:39 INFO - dtype = uint16
10:22:39 DEBUG open(ZarrLocation(/test2.zarr/A/1/0/labels))
10:22:39 DEBUG Created nested FSStore(/test2.zarr/A/1/0/labels, r, {'dimension_separator': '/', 'normalize_keys': False})
10:22:39 WARNING version mismatch: detected:FormatV04, requested:FormatV03
10:22:39 DEBUG Created nested FSStore(/test2.zarr/A/1/0/labels, r, {'dimension_separator': '/', 'normalize_keys': False})
10:22:39 DEBUG treating /test2.zarr/A/1/0 [zgroup] as OMERO
10:22:39 INFO root_attr: multiscales
10:22:39 DEBUG [{'version': '0.3', 'axes': [{'name': 'y', 'type': 'space', 'unit': 'centimeter'}, {'name': 'x', 'type': 'space', 'unit': 'centimeter'}], 'datasets': [{'path': '0'}], 'metadata': {'kwargs': {'axes_names': ['y', 'x']}}}]
10:22:39 INFO root_attr: omero
10:22:39 DEBUG {'id': 1, 'version': '0.3'}
10:22:39 INFO root_attr: channels
10:22:39 DEBUG [{'active': True, 'coefficient': 1, 'color': '0000FF', 'family': 'linear', 'inverted': False, 'label': 'LaminB1', 'window': {'end': 1500, 'max': 65535, 'min': 0, 'start': 0}}]
10:22:39 INFO root_attr: rdefs
10:22:39 DEBUG {'defaultT': 0, 'defaultZ': 118, 'model': 'color'}
10:22:39 DEBUG creating lazy_reader. row:0 col:0
10:22:39 DEBUG img_pyramid_shapes: [(8640, 7680)]
10:22:39 DEBUG target_level: 0
10:22:39 DEBUG get_stitched_grid() level: 0, tile_shape: (8640, 7680)
10:22:39 DEBUG treating /test2.zarr [zgroup] as ome-zarr
10:22:39 DEBUG returning /test2.zarr [zgroup]
10:22:39 DEBUG transforming /test2.zarr [zgroup]
10:22:39 DEBUG node.metadata: {'axes': [{'name': 'y', 'type': 'space', 'unit': 'centimeter'}, {'name': 'x', 'type': 'space', 'unit': 'centimeter'}], 'metadata': {'plate': {'acquisitions': [{'id': 1, 'name': 'single acquisition'}], 'columns': [{'name': '1'}], 'field_count': 1, 'name': 'test', 'rows': [{'name': 'A'}], 'version': '0.3', 'wells': [{'path': 'A/1'}]}}}
10:22:39 DEBUG Transformed: ([dask.array<from-value, shape=(8640, 7680), dtype=uint16, chunksize=(8640, 7680), chunktype=numpy.ndarray>], {}, 'image')
10:22:39 DEBUG ImageSlice.init
10:22:39 DEBUG ImageSlice.init
10:22:39 DEBUG LOADING tile... A/1/0/0 with shape: (8640, 7680)
10:22:39 DEBUG ImageSlice.init`
I hope that this informations could be helpful to better understand the issue.
Thank you very much for your work and for the help in figure out what's going on.
@mfranzon My initial suspicion while looking at the layout and the logs is that the data is stored according to the OME-NGFF 0.3 specification but the chunks themselves are written using the OME-NGFF 0.1 layout i.e. separated by a . separator. Which software generated this Zarr dataset?
In version 0.2 and above, we modified the chunk separator to use / instead. This would explain why you get a blank plane as the library should use a default fill value if no chunk is found. We have a few representative samples for the OME-NGFF 0.3 specification published on this blog post. Probably the most relevant for the modality you are talking about here (HCS, XY dimensions) is the sample generated from idr0094 i.e. https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.3/idr0094A/7751.zarr
@sbesson you are completly right. The problem was exactly this. I generate zarr dataset with a custom implementation, and to generate the chunks I am using dask.to_zarr() function, which probably follows the OME-NGFF 0.1 layout. Do you think it would be better move to another software ?
Another time, thank you very much!
I am not versed enough in the Dask API and the compatibility with the OME-NGFF specification. Quickly looking at https://docs.dask.org/en/stable/generated/dask.array.to_zarr.html and especially the extra keywords delegated to https://zarr.readthedocs.io/en/latest/api/creation.html#zarr.creation.create, it might be possible to pass dimension_separator='/' to write data as nested chunks.
From our side, the other alternative I could suggest is to look at https://pypi.org/project/ome-zarr/ which is the underlying library used by this plugin for reading OME-NGFF but also has some support for writing OME-NGFF. In terms of pros/cons, this library should ensure the best support/compliance with the OME-NGFF versions, the API/functionalities/documentation is probably less mature and still under development but now is actually a good time to get feedback on how to improve this.
For HCS specifically, a starting point might be to look at some of tests https://github.com/ome/ome-zarr-py/blob/ad2c6eb3619b838a5d8607c8521c93a1dde154cb/tests/test_reader.py#L43 which demonstrate the current API semantics for writing a plate.
Thank you very much for your hints! Really useful