increasingly slow searches

[daftscience]

I've run into an issue where searching gets slower over time. I know I may be pushing this TinyDB to it's performance limits. I have a database of around 10k dicts. Each Dict has five items.

So, Here is some psuedo code

class tiny_db():
   def __init__(self):
        self.db = TinyDB('db.json')
        self.table = self.db.table('_default', smart_cache=True)
   def file(self, item):
       self.table.insert(item)
   def locate(name)
       result = self.table.search(where('name') == name)
       return result

Every time I call tiny_db.locate(name) it takes 1 second longer than it did the last time I called. it.
I am probably doing something horribly wrong, Im not entirely sure how the caching middleware works, my grasp of python is pretty weak.

My duct tape fix has been to copy the db into a variable when my class initializes. I'm using that to search. But, I have to insert new items into both. I feel like there may be a better solution.

Sorry for the amateurish question, but thanks in advance.

[eugene-eeo]

The smart cache only caches the same query- i.e. if the name argument passed to tiny_db.locate is always some constant. I don't see why you would hit any performance issues... can you inspect the live table's _cache property and see what you get? With that being said, I think you should try and disable the smart caching to see what happens.

[msiemens]

@daftscience: You're definitely pushing limits here. With 10k elements you should at least consider moving to a more serious database that TinyDB.
That being said, performance across searches should be consistent. I also was able to reproduce this behaviour (10k elements, searching for an element not in the query cache takes 40s, 60s, 120s, 140s, ...). I'll definitely have a look at it!

[msiemens]

Found the offender. Basically, tinydb.utils.catch_warning (which was used in Query.__eq__ on Python 2.x only) failed to properly remove the handlers it added which resulted in O(n) time behaviour.

@daftscience Can you confirm this is fixed in the latest development version?

[daftscience]

Works like a charm. Thanks for the quick response.

[msiemens]

I've now finally released v2.3.0 which includes a fix for this :)