Accelerate the file based storage classes
bdezonia opened this issue · 1 comments
The current file based storage is a second or 3rd generation implementation. It is much faster than it used to be. But I think it can be even faster. The code right now has one buffer it uses. We should make the code configurable so you can specify number of buffers and buffer size and then write a standalone program that tries a few inline algos (like set all values to random numbers and then sum all values), a few random access algos (like Shuffle) and at least one pathological algo (like Reverse.compute(a,a)). It should time those handful of tests and then write a driver that varies buffer size and number of buffers predictably and maybe can converge on the best combination of these things.
Just a note that the file based storage is now using two buffers and that greatly improved speed. But it can be much faster I think with more buffers.