KeyError when using `compressed` for property in StructuredProperty

[LondonAppDev]

Issue

When saving a model with a StructuredProperty that contains a sub property with compressed=True, I get a KeyError for the compressed field.

For example:

from google.cloud import ndb


class MySubModel(ndb.Model):
    data = ndb.JsonProperty(compressed=True)


class MyModel(ndb.Model):
    sub_model = ndb.StructuredProperty(MySubModel)


client = ndb.Client()
with client.context():
    model = MyModel(
        sub_model=MySubModel(data={'test': 1}
    )
    model.put()

The above snippet raises the following exception:

Traceback (most recent call last):
  File "<console>", line 3, in <module>
  File "/usr/local/lib/python3.9/site-packages/google/cloud/ndb/_options.py", line 89, in wrapper
    return wrapped(*pass_args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/ndb/utils.py", line 114, in wrapper
    return wrapped(*args, **new_kwargs)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/ndb/utils.py", line 146, in positional_wrapper
    return wrapped(*args, **kwds)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/ndb/model.py", line 5134, in _put
    return self._put_async(_options=kwargs["_options"]).result()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/ndb/tasklets.py", line 191, in result
    self.check_success()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/ndb/tasklets.py", line 138, in check_success
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/google/cloud/ndb/tasklets.py", line 315, in _advance_tasklet
    yielded = self.generator.send(send_value)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/ndb/model.py", line 5194, in put
    ds_entity = _entity_to_ds_entity(self)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/ndb/model.py", line 748, in _entity_to_ds_entity
    prop._to_datastore(entity, data)
  File "/usr/local/lib/python3.9/site-packages/google/cloud/ndb/model.py", line 4180, in _to_datastore
    prop._to_datastore(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/ndb/model.py", line 2500, in _to_datastore
    value = data[self._name]
KeyError: 'data'

Expected result

I would expect that running model.put() will save and compress any fields within a StructuredProperty.

Environment Details

• google-cloud-ndb version 1.7.3
• Running code inside Docker image python:3.9-slim (host os is macOS 11.2.1)
• Python version: 3.9.1

I've had a look on the existing issues and Stack Overflow and couldn't find any reference to this issue.

Thank you for taking the time to look read this issue. Any suggestions would be greatly appreciated.

[chrisrossi]

Hi, I'm taking a look at this now. I've been able to reproduce and should have a fix shortly. In the meantime, I'm finding that this only happens in "legacy data" mode, the default, which uses the old way of storing structured properties for backwards compatibility with old datasets. If this is a new application, you can pass legacy_data=False to your context to use a newer format for structured properties that doesn't have this issue:

with client.context(legacy_data=False):
     ....

[LondonAppDev]

@chrisrossi thanks a lot for looking into this and for confirming it wasn't wasn't I was doing wrong :) Yes it's a new application so I'll use the legacy_data=False flag.