Add generic stream interface for serialization / deserialization
Opened this issue · 0 comments
Great work @yurymalkov. This is a simple suggestion for a design improvement that I might be able to implement when I have more time. For now I'm creating this issue as a discussion topic.
Currently serialization / deserialization of all internal state is performed via void saveIndex(const std::string &location)
and void loadIndex(const std::string &location, SpaceInterface<dist_t> *s, size_t max_elements_i = 0)
interfaces respectively. This works fine if the user is always comfortable with the state being preserved as a standalone uncompressed file on disk. But there are many scenarios where the user might want to store / retrieve this data within some other file that contains other unrelated data too. Or perhaps they don't want to store it on disk at all but would like to send it over the wire. Or perhaps they want to do some fancy cryptography with it. Or perhaps they want to compress / decompress it. The latter especially makes sense for very large datasets.
The solution to this is to have the aforementioned two interfaces wrap two new public ones that work with std::ostream
and std::istream
streams respectively instead of taking explicit paths to file system paths.
On a related note, the const std::string &location
declarations should probably be migrated to const std::filesystem::path &location
if or when the library is ready to be migrated to C++17. Although I understand that for the time being the intent is to maintain compatibility with C++11.