lebedov/msgpack-numpy

Can't decode encoded structured arrays with subarrays

Closed this issue · 1 comments

egnor commented

Example:

% python3
Python 3.6.8 (default, Jan 14 2019, 11:02:34) 
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'1.17.1'
>>> import msgpack_numpy
>>> s = numpy.recarray((5,), dtype=[('x', 'f', (2,))])
>>> s
rec.array([([5.989970e-38, 0.000000e+00],),
           ([0.000000e+00, 2.847438e-42],),
           ([4.641605e+17, 4.557023e-41],),
           ([2.382207e-44, 0.000000e+00],),
           ([6.726233e-44, 0.000000e+00],)],
          dtype=[('x', '<f4', (2,))])
>>> e = msgpack_numpy.encode(s)
>>> e
{b'nd': True, b'type': [('x', '<f4', (2,))], b'kind': b'V', b'shape': (5,), b'data': b'\x00\x10\xa3\x01\x00\x00\x00\x00\x00\x00\x00\x00\xf0\x07\x00\x00\xf0 \xce\\\x08\x7f\x00\x00\x11\x00\x00\x00\x00\x00\x00\x000\x00\x00\x00\x00\x00\x00\x00'}
>>> msgpack_numpy.decode(e)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/egnor/source/thepisky/local_virtualenv/lib/python3.6/site-packages/msgpack_numpy.py", line 84, in decode
    dtype=np.dtype(descr)).reshape(obj[b'shape'])
ValueError: invalid shape in fixed-type tuple.

This seems to be because of this bit in the code:

# Check if b'kind' is in obj to enable decoding of data
# serialized with older versions (#20):
if b'kind' in obj and obj[b'kind'] == b'V':
    descr = [tuple(tostr(t) for t in d) for d in obj[b'type']]
else:
    descr = obj[b'type']

I'm not sure exactly what compatibility is being maintained, but the tostr(t) turns (2,) into "(2,)" which isn't valid.

egnor commented

My bad -- this is already fixed in the latest version!