mobie/mobie-utils-python

OME-Zarr as file format

Closed this issue · 16 comments

martinschorb commented

Hi,

is OME-Zarr now fully featured if I provide that as format parameter? Anything else I need to consider?
Or can I just run import_image_data to produce OME-Zarr?

constantinpape commented

is OME-Zarr now fully featured if I provide that as format parameter? Anything else I need to consider?

Yes, but so far it only supports v0.3, not v0.4 yet. (I.e. cannot store scale information.) I will look into adding support for v0.4 soon; maybe I have time for this today since it should not be much effort.

Or can I just run import_image_data to produce OME-Zarr?

import_image_data is just the function used internally by add_image_data. So if you pass format=ome.zarr it will currently produce ome.zarr v0.3 data. (But it will only create the ome.zarr container with it's metadata and not write any of the mobie metadata.)

martinschorb commented

Then I might just wait a little until it supports v0.4 and then give it a try.

constantinpape commented

I have implemented v0.4 support now (and in the process removed the bdv.ome.zarr hybrid format, which is not necessary anymore since we can have the scale information in ome.zarr now).
It will take until tomorrow or so since the changes make their way to conda, I will ping you then.

  • 👍1
martinschorb commented

don't worry, my environment use the local nightly clone of this repo ;-) I will give it a try and let you know how it goes...

constantinpape commented

don't worry, my environment use the local nightly clone of this repo ;-) I will give it a try and let you know how it goes...

Ok, but the main changes for this are in https://github.com/constantinpape/cluster_tools. You will need to be on the current master of it for ome.zarr v0.4 support.

martinschorb commented

cool, thanks!

martinschorb commented

Is there a "basic" MoBIE project that "just" generates a bunch of sources from sets of tiff slices using up-to-date functionality?

The scripts for older FIB-SEM projects still seem like older more hacky approaches.

martinschorb commented

I will work my way along the tutorial notebook and create project scripts accordingly. We can then use these as templates for future use.

  • 👍1
constantinpape commented

I will work my way along the tutorial notebook and create project scripts accordingly. We can then use these as templates for future use.

That's great, thanks!

martinschorb commented

Ouch, the reconstruction data format we got from HH is normalized dtype=float32.
Can downscale deal with that? I could not find any type related conversions when browsing the code.

martinschorb commented

Okay, found the answer: no...

File "./mobie/validation/utils.py", line 64, in _assert_true
    raise ValueError(msg)
ValueError: -0.0001567757 is less than the minimum of 0
Failed validating 'minimum' in schema[0]['properties']['imageDisplay']['properties']['contrastLimits']['items'][0]:
    {'maximum': 65535, 'minimum': 0, 'type': 'number'}
constantinpape commented

Ouch, the reconstruction data format we got from HH is normalized dtype=float32.
Can downscale deal with that? I could not find any type related conversions when browsing the code.

Sure, it can deal with float32, no problem.

Okay, found the answer: no...

File "./mobie/validation/utils.py", line 64, in _assert_true
    raise ValueError(msg)
ValueError: -0.0001567757 is less than the minimum of 0
Failed validating 'minimum' in schema[0]['properties']['imageDisplay']['properties']['contrastLimits']['items'][0]:
    {'maximum': 65535, 'minimum': 0, 'type': 'number'}

That's unrelated to the datatype; it's just that negative values cause issues when displaying them in BDV, so we don't allow for them right now. You can either manually set the minimum to 0 (i.e. values smaller 0 will be below the min contrast limit for displaying) or change the data before passing it to mobie-python so that it's minimum is 0.

martinschorb commented

Right, we need to take care of the view later.

As far as I get it, there is no normalisation and dtype conversion. It basically supports whatever vigra or skimage support, correct?

I think it would be ideal to either:
a) enable conversion to bytes or uint16 just before the copy task. Pulling everything out of the import workflow will simply generate data duplicates and compute overhead. I can have a look if you point me to where exactly the copy task is called and try to implement a type conversion.

b) make MoBIE capable of applying a type conversion/intensity remapping on the fly.

It does not make much sense to me if mobie-python supports different data types than the actual MoBIE viewer.
Personally I would be in slight favour of b), but I don't know how that works deep inside BDV core. @tischi

martinschorb commented

Ideally, ideally the supported data types should match exactly OME-Zarr/NGFF.

constantinpape commented

As far as I get it, there is no normalisation and dtype conversion. It basically supports whatever vigra or skimage support, correct?

Exactly.

a) enable conversion to bytes or uint16 just before the copy task. Pulling everything out of the import workflow will simply generate data duplicates and compute overhead. I can have a look if you point me to where exactly the copy task is called and try to implement a type conversion.

b) make MoBIE capable of applying a type conversion/intensity remapping on the fly.

I agree that we should go with one of the two solutions; let's wait what @tischi thinks about it in mobie/mobie-io#85.

It does not make much sense to me if mobie-python supports different data types than the actual MoBIE viewer.

I just want to be precise here: this is not a datatype, but a data range issue. But I agree with you apart from that.

constantinpape commented

This is fixed.