Storing data during on-the-fly image processing

[Flemingjp]

Is there a recommended approach to storing data generated during on-the-fly image processing.

Using metadata seems to be an option, but I am unsure if there is a better way - especially for numpy arrays and loading that data some time after the acquisition has been written to disk

def demo_image_processing_fn(self, image, metadata):
  # Some image processing occurs generating some data
  data = []

  # Save this data (into metadata?)
  metadata['data'] = data

  # Can we discard the image, but not the data?
  return None, metadata

[henrypinkard]

depends what type of data you're generating

If the data is an image, you could have the function return multiple images, both the original and the processed version (though it would have to be unit8 or uint16 I think)
https://pycro-manager.readthedocs.io/en/latest/img_processors.html#returning-multiple-or-zero-images

# Can we discard the image, but not the data?
  return None, metadata

No, Im pretty sure this will cause some kind of error

If you're just trying to save some numpy arrays and don't need your original image data, I would just implement your own data saving with numpy.savez or something similar

[Flemingjp]

Thanks for the advice - in this instance it would be additional non-image data. So I think some own implementation as you suggested is most appropriate.