delayed and partial loading

[oO]

I've started implementing MongoDB support in my Pyramid app with pure pymongo because I didn't like the rigid schema of MongoEngine and MongoKit, but it looks like minimongo is closer to what I need.

However, I do a lot of delayed loading and partial loading of my data, because I store information in the database that I don't usually need to manipulate on a regular basis. Would this be possible with minimongo?

What I'm thinking is something like:

item = MiniMongoObject()
item._id = get_id_from_somewhere()
item.load(fields=['name', 'description'] )

this would partially load the object, only fetching the name and description field
Ideally this would mark the object as partially loaded, so that save() would raise an
error, or use update() behind the scene instead.

I would also like to see MongoObject.mongo_update() take an optional data parameter, so I can use it to send my own field to update ( like '$inc' )

item.mongo_update(  {'$inc": {'counter': 1} }  )

And also expose all the optional update parameters as well ( I sometime need to use safe=True)

[slacy]

I think both these operations are supported by the current API. When you say "MiniMongoObject.collection.find()" you're calling right through to pymongo's find method on that collection, so anything you can do there should work. For example, if you only want to load the 'name' and 'description' fields, it would look like this:

item = MiniMongoObject.collection.find({"_id": id_from_somewhere()}, 
                   {'name': 1, 'description':1})

Similarly, if the MiniMongoObject.collection.update() method is identical to the pymongo update metheod. If you wanted to do an inc on a previously loaded object in "item", as you described, it would look like this:

MiniMongoObject.collection.update({"_id": item._id}, 
                   {"$inc": {'counter': 1}})

I can write up some examples and test cases for these in more detail if you'd like.

[oO]

Thanks, I understand that. what I was hoping for is convenience methods (like the existing item.save() and item.mongo_update() which would save me from typing _{'_id': item.id} everywhere.

You already have some of these on the model itself:

def remove(self):
    """Remove this object from the database."""
    return self.collection.remove(self._id)

def mongo_update(self):
    """Update database data with object data."""
    self.collection.update({'_id': self._id}, self)
    return self

def save(self, *args, **kwargs):
    """Save this object to it's mongo collection."""
    self.collection.save(self, *args, **kwargs)
    return self

So what I'm suggesting is to extend those:

def mongo_update(self, data=None, **kwargs):
    """Update database data with object data."""
    if data == None:
        data = self
    self.collection.update({'_id': self._id}, data, **kwargs )
    return data

I'll try to fork sent a pull request later.

[slacy]

I see, checked out your pull request and commented.

[slacy]

FYI I'm also using Pyramid, and have a nice minimongo-backed session implementation if you're interested. It's pretty raw, but I can put it somewhere and we can collaborate on it if you want .

[oO]

sure. I'm currently using pyramid_beaker with the file store option, so I'm fine either way.
I'm waiting on a fix for pymongo 1.10 which broke pickling ObjectId, and broke the session stuff since I store the current user document in the session for authorization.

BTW, I think I just hosed the users collection on my dev setup running the tests. I was setting the db and collection in a configure call in a file (at the same level as setup.py ), and I think it kept those settings when running the tests. I hope that's what it is and that randomly dropping a collection is not a hidden minimongo feature...

[slacy]

I sure hope minimongo can't result in an accidentally dropped collection! I accidentally dropped my user collection on Friday just because I had some muscle memory and typed "db.user.drop()" instead of db.user.find()" Ugh.

I agree with your point about being able to say:

foo = FooMongoObj(id=some_id).load()

I like that syntax, so that's okay with me. Craft up the unit tests in the pull request and I'll give it a once-over and likely put it in.

[oO]

Btw. this is the part that will drop any model declared within the scope of minimongo, while the test are running... ouch!

def teardown():
    all_models = set(Model.__subclasses__())
    all_models.add(TestDerivedModel)
    all_models.add(TestModelImplementation)
    # all_models.remove(TestModelInterface)
    map(lambda m: m.collection.drop(), all_models)

So even though my model was not using the default test database, it got its collection deleted anyway.

[slacy]

Yikes! I'll be patching and removing that code now. Good catch!

[slacy]

I've just changed the code for this, but thinking about it a bit more:

This code should only drop all models that are imported into the test_model.py scope. So, how could it drop your user classes, which I presume are declared far outside? I guess if you had a modified version of this test or a modified version of minimongo that imported your user class, then it could happen, but I don't see how it could happen otherwise?

Are you sure this is the culprit?

[oO]

since I was working inside of minimongo, I had a file there where I was calling configure and setting it to my database, and a model declaration there. I guess it is in the scope of the module even though it's not inside the minimongo folder.

minimongo olivier$ ls
CHANGES.txt
docs /
minimongo.egg-info
oo_test.pyc
setup.py
README.rst
minimongo/
oo_test.py <-- my class declaration and configure statement is in here...
runtests.py

oO

[slacy]

okay, I'm going to go over your pull request in just a few minutes.

[slacy]

Merged and pushed to master. Thanks!