Create light dataset that can be used as demo with `workflow-calcium-imaging`

Question

Create light dataset that can be used as demo with `workflow-calcium-imaging`

Closed this issue 2 years ago · 8 comments

Answer 1 · 2021-12-21T23:07:06.000Z

Should include scanbox and scanimage as examples

Answer 2 · 2022-01-12T14:59:06.000Z

The scanbox data we have is a single 23GB sbx file. Is it possible to trim it (e.g. down to 3-5GB because that's the case for the scanimage data)?

The only solution that comes to mind for me would be a hacky local instal of sbxreader where you set max frames to your own value.

Answer 3 · 2022-01-12T15:04:21.000Z

The scanbox data we have is a single 23GB sbx file. Is it possible to trim it (e.g. down to 3-5GB because that's the case for the scanimage data)?

The only solution that comes to mind for me would be a hacky local instal of sbxreader where you set max frames to your own value.

I wasn't very clear in my question. I'm able to read the data and it has a shape of (7530, 4, 1, 512, 796). The first number in the list is the number of images. The rest shows the plane number, dimensions, etc. If I slice this matrix to get (500, 4, 1, 512, 796), would that be appropriate? Am I breaking anything? Are there any raw data that depends on this? Is 500 images enough to test our workflows?

Answer 4 · 2022-01-12T15:13:36.000Z

Am I breaking anything? Are there any raw data that depends on this?

I don't know for sure. If you're able to save and reload, I think it would we worth running the light data through the imaging schema. Kabi or Dimitri could then take a look to see if the results look on-part with having a truncated dataset.

Answer 5 · 2022-01-12T15:41:15.000Z

Thanks @tdincer @CBroz1. I have not worked with Scanbox files enough to know the nuances. Suite2p requires at least 200 frames (see suite2p/registration/register.py). Not sure if CaImAn has a minimum requirement. If this light dataset will be for the task_mode=trigger demo and pytests, I don't see an issue with slicing the dataset and resaving. If this light dataset will also be used for the task_mode=load, then you will have to rerun Suite2p or CaImAn because the processed data that is hosted on djarchive should correspond to the same raw data.

Answer 6 · 2022-01-12T15:55:03.000Z

Updated my response above.

Answer 7 · 2022-01-12T16:26:52.000Z

Thank @kabilar and @CBroz1! I did some search but couldn't find a tool to write data into a sbx file. Suite2p or other libraries always convert the data into other formats. So, looks like, we won't have a light scanbox example for now.

Answer 8 · 2023-02-23T15:18:22.000Z

We have now created a light dataset for ScanImage and Suite2p. On djarchive, see workflow-calcium-imaging-data-v2. We do not currently have a need for the other data formats.