irmen/Pyrolite

java: pickling calendar results in invalid pickle for Python 3.x

irmen opened this issue · 4 comments

Commit 9448bce broke the Calendar pickling to datetime objects.

Pickling this calendar object:

    cal = Calendar.getInstance();
    cal.set(Calendar.YEAR, 2015);
    cal.set(Calendar.MONTH, 4);
    cal.set(Calendar.DAY_OF_MONTH, 18);
    cal.set(Calendar.HOUR, 23);
    cal.set(Calendar.MINUTE, 59);
    cal.set(Calendar.SECOND, 59);
    cal.set(Calendar.MILLISECOND, 0);
    cal.setTimeZone(TimeZone.getTimeZone("Europe/Amsterdam"));

results in a pickle that makes python 3.x crash when unpickling:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xdf in position 1: ordinal not in range(128)

(works ok in python 2.x)

cause: pickle contains SHORT_BINSTRING opcode which fails decoding in Python3. Python 3 pickles it with a SHORT_BINBYTES opcode instead.

(note: the SHORT_BINSTRING opcode is also used when pickling array.array, but it only encodes the typecode character there and it seems to work fine when unpickling in python3)

@airhorns for the commit above, you mentioned it was needed to be able to add a timezone to the constructor tuple. I don't really see that happening anywhere. Was this change actually valid?

@airhorns I took the liberty of reverting the serializer code for Calendar back to the old 7-arg constructor one. I adapted the DateTimeConstructor to accept an 8th argument, being the tzinfo object. This seems to work fine, both with python 2 and 3. Unfortunately I discovered a new issue #26

Damn, I should have tested on Python 3, my apologies. Python 3 and Python 2 work with an 8 argument tuple?

Yes, it seems they both work now.