Performance Bottlenecks

[mikeal]

The initial implementation has been written for clarity and to prove that everything works. As a result a lot of the implementation has known performance bottlenecks that I want to write down and eventually go back to fix.

• Unnecessary async.
  • Whenever we have a node in cache we can avoid adding anything into the event system if the code calling it doesn’t use await and instead uses the promise API. My initial implementation of CADB saw a 30x perf increase from this sort of work but the code became too complex to debug for a proof-of-concept. Any attempts at this kind of perf work were table for IPSQL in order to avoid the complexity, but at some point we’ll have to eat it for the perf benefits.
• Storage Formats
  • Something like CADB will be faster than leveldb and possible even CAR export.
• Codec
  • DAG-CBOR can be improved upon. If ZDAG were to become stable it would reduce the block size of a lot of the indexes by as much as 50%.
• Too many serial read operations.
  • There’s a few places where we’re using generators that would be much faster (but consume more memory) if they were converted to parallel recursive async function calls.