googleapis/python-ndb

PickleProperty put in Python 2.7 GAE NDB can't be read by Cloud NDB on Python 3 if it contains certain types

Tony-Phillips05 opened this issue · 1 comments

Discovered while migrating to python 3

if type(value) is bytes: # pragma: NO BRANCH
return pickle.loads(value, encoding="bytes")

Related to #595

The current PickleProperty referenced in the github code link above uses encoding="bytes" when unpickling. This does not work for all data types that was previously pickled with NDB in python 2.7. According to: This post, if you had pickled objects that contained datetime and some others, the only way to unpickle correctly is to use encoding="latin1"

We have internally fixed this with:

`class PicklePropertyFixed(model.BlobProperty):

def _to_base_type(self, value):
    return pickle.dumps(value, pickle.HIGHEST_PROTOCOL)

def _from_base_type(self, value):
    """Because the google.cloud.ndb.model.PickleProperty ONLY decodes using the
    'bytes' encoding, we must use this custom class to override this.
    The reason is because the encoding 'latin1' has to be used if the pickled object
    has instances of datetime, date, time, or numpy arrayswhich. Our model uses datetime
    """
    if type(value) is bytes:
        return pickle.loads(value, encoding="latin1")
    return pickle.loads(value)`