ilastik/lazyflow

BlockwiseFileset: Avoid creating empty files

stuarteberg opened this issue · 0 comments

The BlockwiseFileset class creates a lot of small or empty files that are used for bookkeeping purposes. On HPC storage, metadata performance on such files introduces a large overhead. Somehow these book-keeping items need to be kept somewhere else. Probably a simple database or sqlite store would be fine, provided it can handle a bit of concurrency.