omnilib/aiosqlite

[Question] Why not use ThreadPoolExecutor?

Closed this issue · 2 comments

Before I found this library, I was using the stdlib sqlite3 module in this kind of fashion:

with ThreadPoolExecutor(max_workers=1) as executor:
    loop = get_running_loop()
    connection = await loop.run_in_executor(sqlite3.connect, ...)
    cursor = await loop.run_in_executor(connection.cursor)
    await loop.run_in_executor(cursor.execute, ...)

I wrapped a lot of these common functionalities in a quite similar way to your module here (but much less general, as it was a means to an end), but it all worked quite well, with the ThreadPoolExecutor always limiting to one executing thread (in practice, always the same thread).

I've noticed that this module replicates a lot of the same behavior of ThreadPoolExecutor and run_in_executor. Is there a functionality reason for this, or is it just a matter of personal choice? I'm wondering if the other places I still am using the ThreadPoolExecutor for the purpose of dedicating a single background thread in an async application is susceptible to some particular fatal flaw.

The easy answer is that back when I first wrote the module, ThreadPoolExecutor wasn't well typed, and the semantics of run_in_executor were still a bit shaky, and a hand-rolled thread was easier to reason about in regards to ensuring a connection was always used with the same thread.

(in practice, always the same thread).

The thread name is the same, but if something happens and the thread gets recycled, then IIRC the connection is no longer valid.

That's a good point. SQLite should work either way, as long as it's only used from one thread at a time, but check_same_thread would probably bark if enabled. Given the extra unneeded functionality of ThreadPoolExecutor (and the fact that it makes no API guarantees that you want, like always using the same thread), I can respect the decision to roll your own.

Thank you for responding.