More efficient buffering algorithms
Opened this issue · 0 comments
ljwoods2 commented
- Setup ASV benchmarks to get a baseline of performance in current solution
- Experiment to find the fastest on-average fixed chunk size
- In reader, use trajectory slicing to pull the next needed chunks rather than allowing LRU cache to handle it
- In writer, experiment with parallelizing writes across different datasets (IE step, time, etc) to see if engaging more cores improves performance in IO operations