Export as archive or sharded file type
Closed this issue · 1 comments
Hi! It was nice to meet everyone at the HBP summit.
Webilastik is working well, however one difficulty I have is that when exporting precomputed (or any tiled format) images it is very slow to upload them to the bucket (due to the large number of files). It also introduces further penalties when I import them into down stream apps for further processing (again because of the slowness of copying so many chunks). I have a number of solutions and am interested on hearing your opinion.
-
Likely the best solution, Neuroglancer actually supports a fileformat which combines chunks into few larger files precisely because people run into similar issues. https://github.com/google/neuroglancer/blob/master/src/neuroglancer/datasource/precomputed/sharded.md I havent tested the sharded file format myself but it sounds optimal. Especially since neuroglancer can still stream chunks from it without any modifications. If we could export from Ilastik in this format this would be great.
-
Simply exporting segmentations as a png would work well for us but is probably not optimal since webilastik wouldnt be able to read and display them. So this would work for us (maybe as a short term solution) but probably isnt optimal in the long run.
-
zipping the precomputed file prior to export. This would also work however has many of the same drawbacks as point number 2.
Thank you in advance and I hope there is an easy fix to this issue!
all the best,
Harry