Speed up augur filter
Opened this issue · 0 comments
victorlin commented
Filtering and subsampling logic in augur filter is done using pandas DataFrames to represent input metadata. Due to the need to load DataFrames into memory, this has been optimized to work on large datasets (e.g. SARS-CoV-2) by loading DataFrames in chunks since the entire dataset is too large to load on a typical machine's RAM. Compared to a non-chunked approach, this is less efficient (requires multiple passes through the metadata) and less intuitive (requires workarounds and extra variables to coordinate operations across chunks).
There are two ways to tackle this. See individual issues for details:
TODO: @victorlin to add more about sequence I/O