Streaming objects from Mongoose is very slow, and sometimes runs out of memory

[jwalton]

Detailed reproduction instructions here:

https://gist.github.com/jwalton/6575901

But basically, if longjohn is present, then trying to stream 40K records from Mongoose will run well, but as you get close to 12K objects (YMMV) things start slowing way, way , way down, and then you start getting objects one or two every ten seconds, and the memory footprint of node on my mac jumps up to 1.5G. In a real production app, I've seen the node VM run out of memory and die.

If you take longjohn out of the equation, then things run a lot faster (including creating objects), everything finishes OK, and the memory footprint never gets up much above 75MB.

Interestingly, if you modify this app like this:

# Line X
cursor = B.collection.collection.find()

and this:

# Line Y
object = new B object

then the program completes successfully regardless of whether you have longjohn enabled or not, which is weird, since you're effectively doing the same thing. There must be something specific about the way Mongoose streams objects that makes this happen.