Numpy format (*.npy) serialization and deserialization.
NPY is a simple binary data format. It stores the type, shape and endianness information in a header, which is followed by a flat binary data field. This crate offers a simple, mostly type-safe way to read and write *.npy files. Files are handled using iterators, so they don't need to fit in memory.
To use npy-rs, two dependencies must be specified in Cargo.toml:
npy = "0.3"
npy-derive = "0.3"The second dependency implements the custom derive macro. A typical way to import everything needed is:
#[macro_use]
extern crate npy_derive;
extern crate npy;Several usage examples are available in the examples directory; the simple example shows how to load a file, roundtrip shows both reading and writing. Large files can be memory-mapped as illustrated in the large example.
Version 0.3 brought ten-fold performance improvements. On my laptop, it now loads and writes files from a ramdisk at approx. 700 MB/s.
Only the header is parsed on the NpyData::from_bytes call. The data can then be accessed sequentially by iterating over NpyData, randomly by using the get function, or the whole file can be deserialized into a Vec at once by using the to_vec function. Only the third option requires the whole file to fit into the RAM at once.