googleapis/python-ndb

Document that this project is in maintenance mode?

snarfed opened this issue · 14 comments

snarfed commented

From googleapis/google-cloud-python#10566 (comment) :

The current owners of the Datastore libraries team let me know that NDB is not currently being developed, but things may change in the future.

OK! Fair enough. Maybe consider documenting that in the README and on https://googleapis.dev/python/python-ndb/latest/ ? And also whether it's maintained, and just not actively adding features, or totally unmaintained?

(Thank you for all of your work up until this point, btw!)

meredithslota commented

Hi @snarfed — thanks for filing this issue. Agreed that projects should clearly communicate their level of support. We're doing our planning for the next couple of quarters now and will follow-up.

  • 👍4
jacobg commented

I'm sad to read that NDB won't be actively updated moving forwards. NDB is a highly productive library with useful higher level abstractions, such as typed data models, conjunctive/disjunctive queries, etc. Whereas the Python Datastore library is a lower level library that requires a lot more app code for parallel queries, in-memory filtering/merging, type validation, etc. Both have their use cases.

Even if NDB is pretty mature and doesn't need major new features, we still need its dependencies to be actively bumped in order for it to keep up with the latest ecosystem.

  • 👍5
jacobg commented

I briefly took a look at what it would take to bump dependencies, i.e., edit setup.py:

        "google-cloud-datastore >= 2.7.0, < 3.0.0",
        "pymemcache",
        "redis",
        "pytz"

It's not quite a trivial upgrade. The whole way the google-cloud-datastore generates code stubs changed:
https://github.com/googleapis/python-datastore/blob/main/UPGRADING.md

There's a lot of calls in the ndb code directly to proto and grpc functions that has totally changed. For me, I'd have to delve into how it all works, which I haven't dealt with before. There'd probably be value in making a community effort, so it's not just every individual doing this privately.

Has anyone attempted this?

  • 👍3
jacobg commented

I went ahead, and bumped to google-cloud-datastore 2 on a forked repo here:
https://github.com/jacobg/python-ndb/tree/datastore-v2

Here are the commit notes:

[Release 2.0.0:]
Bump to google-cloud-datastore v2;
Drop Python 2.7 and 3.6 (match versions supported by GAE/GCF runtimes);
Default to Python 3.9

All unit and systems tests pass on my dev computer. I've also updated my own business applications to use it, and there are no known issues.

Here are a few extra thoughts:

  1. It would be great if Google were to merge and take on maintaining this v2 branch. It could even be a non-main v2 branch on this repo, as long as they merge future commits into it so that it stays up to date. Or if anyone has suggestions to make this branch a community repo so that the responsibility of maintaining it does not fall upon me.
  2. It would great to be able to enable CI on GitHub to run all tests for this fork. It seems that Google has their own internal build system, and they just clone the source code to GitHub without using GitHub for CI.
  3. Cloud Datastore recently released native support for !=, IN, and NOT_IN query filters for Firestore in Datastore mode (legacy Datastore not supported). It would be great to take advantage of that NDB to avoid multiple queries where possible. -- https://cloud.google.com/datastore/docs/release-notes#June_17_2022
snarfed commented

@jacobg oh wow, awesome, thank you for doing this! I tried it out and ran the tests for a handful of my projects, ~2k total, and ran a couple of the apps locally and poked at them interactively, and everything passed and seems fine.

Re your extra thoughts, fully agreed with 1 and 3. For 2, I assume you mean the ndb unit tests? Looking at an example PR eg #778, doesn't look like they're run automatically in CI. Maybe try setting up and running them locally?

jacobg commented

@snarfed With regards to CI, what I means is we should configure Travis CI or something like that so that unit tests automatically run when a commit is made. And shows badges on the README. It does seem Google has some of its own build hooks in this repo, but not sure if we would use that for an independent repo, and it doesn't seem they run unit tests.

  • 👍2
meredithslota commented

Hi @snarfed — thanks for filing this issue. Agreed that projects should clearly communicate their level of support. We're doing our planning for the next couple of quarters now and will follow-up.

Development efforts are ramping back up after a quiet period; thanks for your patience here (and thanks @rwhogg for your recent contributions!).

  • 🎉3
meredithslota commented

Dependency updates pending in #841

  • 👍1
  • 😄1
  • 🎉1
  • ❤️1
  • 🚀1
rwhogg commented

@zyzniewski the library documented on that page is the App Engine NDB library (source code). It's an older version of this project and we recommend you use this one, which is the Cloud NDB library.

The most important differences are:

  1. The App Engine NDB library does not support Python 3
  2. The App Engine NDB library only supports the App Engine environment

Whereas Cloud NDB supports Python 3 (some older releases also supported Python 2) and will work both within and outside the App Engine environment.

zyzniewski commented

@rwhogg I'm not referring to datastore-ndb-python but appengine-python-standard ("Google App Engine bundled services SDK for Python 3" https://github.com/GoogleCloudPlatform/appengine-python-standard) - which also provides ndb

rwhogg commented

Ah, sorry for the confusion. That is also a version of the App Engine NDB library that I mentioned above.

We recommend you use this repo's library rather than the one there.