Metadata system tests flaky

[chrisrossi]

Got what appears to be a transient test failure in Kokoro with PR #696, involving one of the metadata tests:

________ test_get_properties_of_kind_different_namespace[DiffNamespace] ________

dispose_of = <function delete_entity at 0x7f68eb5fdb50>
namespace = 'DiffNamespace'

    @pytest.mark.usefixtures("client_context")
    @pytest.mark.parametrize("namespace", ["DiffNamespace"])
    def test_get_properties_of_kind_different_namespace(dispose_of, namespace):
        from google.cloud.ndb.metadata import get_properties_of_kind

        class AnyKind(ndb.Model):
            foo = ndb.IntegerProperty()
            bar = ndb.StringProperty()
            baz = ndb.IntegerProperty()
            qux = ndb.StringProperty()

        entity1 = AnyKind(foo=1, bar="x", baz=3, qux="y", namespace="DiffNamespace")
        entity1.put()
        dispose_of(entity1.key._key)

        properties = eventually(
            lambda: get_properties_of_kind("AnyKind"), _length_at_least(4)
        )

        assert properties == ["bar", "baz", "foo", "qux"]

        properties = get_properties_of_kind("AnyKind", start="c")
>       assert properties == ["foo", "qux"]
E       AssertionError: assert [] == ['foo', 'qux']
E         Right contains 2 more items, first extra item: 'foo'
E         Full diff:
E         - []
E         + ['foo', 'qux']

tests/system/test_metadata.py:248: AssertionError

Testing the metadata is tricky because of eventual consistency. This failure is particularly odd, though, because first we wait for the get_properties_of_kind call without the start argument to return the right number of properties, so presumably the database has reached an internally consistent state. But then calling get_properties_of_kind with the start parameter, we're back to not finding any properties. Is it possible the first call hit a backend resource that had been udpated with the latest metadata, but then the next call hit a different backend resource that still hadn't been updated?

We might need to look for this pattern in all the metadata tests and make sure all metadata calls use eventually, not just the first call in a test.