BlockwiseFileset: Avoid creating empty files
stuarteberg opened this issue · 0 comments
stuarteberg commented
The BlockwiseFileset
class creates a lot of small or empty files that are used for bookkeeping purposes. On HPC storage, metadata performance on such files introduces a large overhead. Somehow these book-keeping items need to be kept somewhere else. Probably a simple database or sqlite store would be fine, provided it can handle a bit of concurrency.