Write to a mat file without loading into memory
Closed this issue · 5 comments
Is there any way to append data to a .mat file without loading the entire file into memory? I am specifically thinking of an example where we continually add to the end of a Struct object.
In Matlab you can do something similar using.
m = matfile('myFile.mat','Writable',true);
m.y(81:100,81:100) = magic(20);Any guidence much appreciated. Thanks!
It's possible to append entries/variables at the root level, but structs are written as a single root-level variable with a size prefix.
By default the data is also compressed, so I can't think of a way to do this without rewriting the whole struct and adjusting at least the file offsets of the other variables. I'd be very surprised if MATLAB could do it differently under the hood.
Btw. can you provide more details about your use case? There may be other ways to solve it better.
Sure. I have a large list of objects with specific data fields (e.g. waveform, serial number, time stamp etc) that I want to write to a .mat file. Ideally I'd like to write a .mat file that is around 1 GB in size before starting a new file. I could hold a list of the objects, convert them to a Struct and then write them all to a .mat file but that will take up substantial amount of memory and likely make the program I am using unstable. Another option is to load the data sequentially, i.e. load a subset of objects into memory, stream those to a .mat file, load the next section, append to the .mat file etc.
There may be no good way of doing this which is fine but if there was that would be great (fantastic work on MFL by the way - I use it all the time). Thanks for the help.
I'd try appending root-level entries and then combining them into a struct via structData = load('logFile') in MATLAB. You can then iterate over all entries via fields(structData). The file size limit is actually only a per-variable limit, so you can have >10GB files as long as each entry is less than 2GB.
Here is an example for incremental append writes: Mat5Examples.java#L251-L282
Thanks for the feedback 🎉Let me know in case you ended up writing any common utilities/helpers that would be good to integrate into the main library.
Apologies for the late reply and thanks very much for the example - it looks as if this allow users to write additional variables to a mat file but not append to an existing array (which I always thought was a long shot). I can defintely work with this and it helps a lot with data management. Thanks again for a great library!