MongoEngine counts zero objects in non-empty collection
arielpontes opened this issue · 8 comments
I have a Mongo collection with 16 objects. When I count the objects directly, I get the expected result:
from mongoengine.connection import get_connection
client = get_connection()
db = client["mydb"]
col = db["users"]
col.count_documents({})
# 16
However, if I try to count using the mongoengine ODM, I get zero:
from app.documents import User
User.objects.count()
# 0
How is this possible? What could I be doing wrong?
Send me a reproducible snippet, my guess is that you modified the model (maybe modified the inheritance) after you inserted records
I didn't "modify" the document after inserting records, I created the document after there were already records. This is an existing project that was using PyMongo to do inserts manually, and now I'm trying to slowly migrate the code to use mongoengine.
I will try to make a simplified snippet, but in the meantime I can tell you it's a tricky document with some nested data, and sometimes the nested data fields are different depending on the object. Could this be the problem? In any case, I would expect the "count" to be correct regardless of mismatching fields, and I would expect to get errors only when I try to access the mismatching fields in my instances. And even if the count is wrong, I would at least expect to get an error or warning explaining why this might be the case. Are my expectations unrealistic? Am I missing something about how mongoengine works?
Update
Turns out I only get this problem when connecting to our dev database:
import mongoengine as me
client = me.connect("mydb", host="mongodb+srv://myuser:mypass@myhost.com/admin?retryWrites=false&replicaSet=atlas-13i89g-shard-0&readPreference=primary&srvServiceName=mongodb&connectTimeoutMS=10000&authSource=admin&authMechanism=SCRAM-SHA-1")
db = client["mydb"]
col = db["people"]
col.insert_many([{"foo": "bar"}, {"num": 13}])
print(f"Manual count: {col.count_documents({})}")
class Person(me.Document):
meta = {"collection": "people"}
print(f"ODM count: {Person.objects.count()}")
Expected results
Manual count: 2
ODM count: 2
Actual results
Manual count: 2
ODM count: 0
When I connect to my local mongo (client = me.connect("mydb", host="mongodb://local:local@mongo:27017")), I don't get this problem at all.
If you modify the Model (I e the Mongoengine Schema), you must apply migrations. My guess is that in your case you have inheritance set up on the User model but something changed in the inheritance path (e.g the parent class of User got renamed). This is described here https://docs.mongoengine.org/guide/migration.html#example-2-inheritance-change
This is the only case that I am aware of that could lead to that issue. (Assuming you connected to the same database of course)
Look at your stored documents, and at the _cls attribute if any, this is where Mongoengine stores the class name
My User document doesn't use inheritance. It looks something like this:
class DjangoUser:
"""
Django expects the logged in user attached to requests to inherit from
django.contrib.auth.models.AbstractBaseUser. Since we instead have a User
document corresponding to an object in a Mongo collection, we have to make
sure we respect certain basic aspects of the AbstractBaseUser inteface.
"""
is_authenticated = True
def get_username(self):
return self.username
class User(DjangoUser, me.Document):
email = me.EmailField()
details = me.EmbeddedDocumentField(UserDetails)
# ...
meta = {"collection": "users"}
As I've mentioned, I didn't "modify" the document because there was no document. The collection existed before I introduced MongoEngine into the project. I didn't see the _cls attribute being added in any of my experiments.
.count() works so something in your implementation or in your existing collection's documents is messing up.
Double check that the pymongo collection used by User is the expected one , it can be accessed with pymongo_coll = User._get_collection().
You can also increase verbosity of your mongo instance logs to check the query. Or configure pymongo to log the queries (recipe is in Mongoengine docs)
Unless you can come up with a reproducible snippet, there s not much more I can do to help ...
I think I figured out the problem. The connection string I was using to connect to the dev db included a database name (admin), and that was overriding the mydb I was passing to connect as the first parameter. I removed that and now everything works as expected.
I still find it weird though that PyMongo and mongoengine were behaving differently even though I used the same connection to make the direct query. Even if I connected incorrectly to the admin database when I call the connect function, shouldn't the line db = client["mydb"] give me the db I expect?