int-brain-lab/ibllib

mesoscope extractor expects nr of FOVs == nr or ROIs in raw SI tif

Closed this issue · 6 comments

For multi-depth imaging, the nr of FOVs we define in the metadata (one per unique scanfield*depth) is not equal to the nr of 'enabled' ROIs in the header of the scanimage tifs (one per unique scanfield). However, the mesoscope extractor currently asserts this and returns an error if it doesn't, causing MesoscopeSync task to fail. This assertion should either be removed or replaced with something more general referring to the unique roiUUIDs e.g. something like
assert sum(x['enable'] for x in artist['RoiGroups']['imagingRoiGroup']['rois']) == len(np.unique(raw_imaging_meta['FOV']['roiUUID']))

assert sum(x['enable'] for x in artist['RoiGroups']['imagingRoiGroup']['rois']) == len(FOVs)

NB the roiUUID field was added to the metadata a while ago but there are older sessions for which this field doesn't exist

Example sessions where this bug happens:
SP048/2024-02-05/001
SP048/2024-02-07/001
SP048/2024-02-08/001
SP048/2024-02-09/002
SP048/2024-02-12/001
SP048/2024-02-13/001
SP048/2024-02-14/001
SP048/2024-02-16/001
SP048/2024-02-23/001

k1o0 commented

I've set all the above sessions to waiting.

@k1o0 Thank you, but the tasks are returning exactly the same error. It seems to be running the old version of the code.

2024-06-20 14:17:18,998,998 INFO [tasks.py:196] Starting job <class 'ibllib.pipes.mesoscope_tasks.MesoscopeSync'>
2024-06-20 14:17:18,998,998 INFO [tasks.py:199] running ibllib version 2.35.2
2024-06-20 14:17:18,999,999 INFO [tasks.py:378] Checking input files
2024-06-20 14:17:19,036,36 INFO [tasks.py:313] All input files found: running task
2024-06-20 14:17:19,036,36 INFO [tasks.py:204] Setup value is: True
2024-06-20 14:17:19,337,337 ERROR [tasks.py:224] Traceback (most recent call last):
File "/home/ibladmin/Documents/PYTHON/envs/iblenv/lib/python3.10/site-packages/ibllib/pipes/tasks.py", line 221, in run
self.outputs = self._run(**kwargs)
File "/home/ibladmin/Documents/PYTHON/envs/iblenv/lib/python3.10/site-packages/ibllib/pipes/mesoscope_tasks.py", line 663, in _run
_, out_files = mesosync.extract(
File "/home/ibladmin/Documents/PYTHON/envs/iblenv/lib/python3.10/site-packages/ibllib/io/extractors/base.py", line 54, in extract
out = self._extract(**kwargs)
File "/home/ibladmin/Documents/PYTHON/envs/iblenv/lib/python3.10/site-packages/ibllib/io/extractors/mesoscope.py", line 638, in _extract
_, fov_time_shifts, line_time_shifts = self.get_timeshifts(imaging_data['meta'])
File "/home/ibladmin/Documents/PYTHON/envs/iblenv/lib/python3.10/site-packages/ibllib/io/extractors/mesoscope.py", line 770, in get_timeshifts
assert sum(x['enable'] for x in artist['RoiGroups']['imagingRoiGroup']['rois']) == len(FOVs)
AssertionError
k1o0 commented

That’s unfortunate. I merged the changed into the multiEnv branch which whiterussian should be using. I’m sans computer until Tuesday but perhaps @chris-langfield could take a look with the help of @mayofaulkner as I know you’d like these extracted soon

k1o0 commented

Appears to have re-run without error