Get Matlab Cell
simoneb opened this issue · 20 comments
Apologies for the probably noob question but I'm quite unfamiliar with the data format, I'm just trying to read a Matlab file from Node.js. The file has a property which is of type Cell with certain dimensions, say 15x182. It can be easily displayed in Matlab and looks like a spreadsheet containing various data types, including numbers and strings.
I'm trying to find a way to get that data using this library. I've been able to open the file and read its contents, but when I try to access that attribute, all I'm getting back is some sort of numerical representation of the content that I can't easily make sense of.
Here's the program:
const f = new h5wasm.File(filePath);
const prop = f.get('MyProp');
console.log(prop.attrs);
console.log(prop.metadata);
console.log(prop.json_value);
And here's what it prints:
{ MATLAB_class: [Getter] }
{
signed: false,
type: 7,
cset: -1,
vlen: false,
littleEndian: false,
size: 8,
shape: [ 15, 182 ],
maxshape: [ 15, 182 ],
chunks: [ 15, 182 ],
total_size: 2730
}
Uint8Array(21840) [
160, 10, 0, 0, 0, 0, 0, 0, 8, 12, 0, 0,
0, 0, 0, 0, 104, 13, 0, 0, 0, 0, 0, 0,
224, 14, 0, 0, 0, 0, 0, 0, 72, 16, 0, 0,
0, 0, 0, 0, 184, 17, 0, 0, 0, 0, 0, 0,
32, 19, 0, 0, 0, 0, 0, 0, 128, 20, 0, 0,
0, 0, 0, 0, 176, 23, 0, 0, 0, 0, 0, 0,
24, 25, 0, 0, 0, 0, 0, 0, 120, 26, 0, 0,
0, 0, 0, 0, 216, 27, 0, 0, 0, 0, 0, 0,
168, 30, 0, 0,
... 21740 more items
]
Can you attach an example file to this issue so I can have a look?
Sure thing, here you go. Let me know if you need anything else, and thanks a lot for your help.
some list.zip
This seems to describe how values are encoded from MATLAB into an HDF5 file: https://pythonhosted.org/hdf5storage/storage_format.html
In the file above, the dataset of interest seems to be at '/StratList', and it has the Reference datatype, shape = (15,2). It has an attribute "MATLAB_class" which has value "cell". Each value of this dataset is a reference to another object, in this case all of them are datasets in the '/#refs#' group at the root level. In that group there are 31 datasets named ["a", "b", "c" ... "A", B", "C", "D", "E"]. Of those, some are of type '<f8', (float64) while some of are of type '<H' (uint16). For example, dataset "/#refs#/b" has dtype uint16 and these attributes:
{"H5PATH": "/#refs#/b", "MATLAB_class": "char", "MATLAB_int_decode": 2}
This indicates that it is a string type in MATLAB, and should be interpreted as a utf-16 string ('some other report')
Unfortunately, I had to use h5py or h5dump to do this investigation, as support for Reference data is not built into h5wasm at this time. Even if this support was added, you would still have to build some post-processing tools in order to interpret the cell data (since e.g. strings are recorded as uint16 arrays with 2 dimensions)
Support for references would take a little while to build - there are multiple different types of reference, and you don't know which kind each reference is without calling the HDF5 API to check it. So you could have a dataset of references that are a mixture of object references (any of file, group, dataset, named datatype, or attribute) and dataset region references.
Thanks for looking into it, and wow, I never thought it would be so intricate to figure out what's in that file, considering that opening it with matlab is a single line of code. I guess I don't have many options at the moment besides keeping using matlab, right? There's no easy way to read these files otherwise
Matlab is storing Matlab-specific hints for how to interpret the data in the attributes, so their own interface is able to seamlessly read/write these files. It wouldn't be too hard to write a post-processor in javascript for a limited number of cases - for the most part h5wasm already knows how to handle all the datatypes in https://pythonhosted.org/hdf5storage/storage_format.html#how-data-is-stored - the special case seems to be strings.
So do you think that based on the analysis you shared before I should be able to figure out how to get the strings from the file?
Yes, there are functions in javascript that make it straightforward, once you know the data is utf-16 - e.g.
new TextDecoder('utf-16').decode(new Uint16Array( [115, 111, 109, 101, 32, 111, 116, 104, 101, 114, 32, 114, 101, 112, 111, 114, 116] ))
// 'some other report'Still, I would first need to add support for References to h5wasm. I'm not sure when I'll be able to do that.
Oh ok, so there are some missing features that wouldn't make it possible at the moment, right?
Yes - but I'll update this ticket when I implement support for References.
Thanks a lot 👌
Ok, this is now possible with the newly published v0.7.1 h5wasm. See: https://observablehq.com/@bmaranville/reading-matlab-hdf5-files-with-h5wasm
Thanks @bmaranville , I'll try it out!
I see errors at the link you shared though:
Apologies - getting a notebook set up so all the async bits run in the right order is a little tricky. I edited the notebook - if you reload, it should be working.
Nice, I can see that working now! Is there a way to tell the dimensions of the cell so I know how long each row is? Because at the moment it's just a flat array
Of course - the shape is stored in the metadata and is available as dset.shape, which returns [15,2]. You can also see it in the expanded metadata in the notebook:
That's great, thanks a lot for this!
Feel free to close this issue of course!
@bmaranville I've come across another file which I cannot seem to be able to parse correctly. Can I send it to you privately?
The file is generated using the same Matlab software that I used to generate the one I shared previously, but I cannot explain why the file size in this case is extremely reduced compared to the previous one. It seems like the same Matlab software I'm using, at some point and for reasons I cannot explain started generating this file in a different format, and a much smaller size. I don't have Matlab but using Matlab online I can open both files just fine, and I can't explain the difference in size.
Thanks!
Sure - feel free - my email address is public in my profile.