Load data from OMF files on demand
Opened this issue · 2 comments
The implementation of loading OMF v2 files (#36) currently only supports loading either all the binary or none of the binary. However, the ZIP format could support loading data on demand. This could look a few different ways:
- specify the elements you want loaded and only load binary for those
- load binary on demand, when first accessed
- read directly into the OMF file to get binary data
All of these would improve usability, and (3) could significantly reduce the memory footprint.
This follows from the conversation here: https://github.com/gmggroup/omf/pull/91#discussion_r322523415
Resolves #72 ? At least this is what I wanted from that issue!
Personally I would lean towards option 2 (or a mix of them).
When reading an OMF file in "overview" mode the JSON parts of the elements are loaded however binary parts, which should be the heaviest parts, are kept in an uninitialized state until the data is accessed. Option 1 could be combined with this to just force load binary parts on some elements.
The (possibly minor) downside of all this is that the input file must be kept open for the lifetime of any of the objects in the project. There may be more issues we expose ourselves to with this such as not being able to overwrite the input file in place.
A benefit for this is when writing an output file uninitialized binary parts can have their compressed entries copied directly from the still active input file.
Option 3 sounds good but I'd be worried about repeat read/write performance unless we cache stuff in memory. It then becomes more or less what I would expect option 2 to be doing.