waku-org/nwaku

chore: optimize store v3 queries

Opened this issue · 0 comments

Background

It's been detected that store-v3 queries don't respond as quickly as store-v2 and we need to understand why.

Details

This has been originally reported by @richard-ramos . I add some notes shared by him:

./build/query \
   --cluster-id=16 \
   --pubsub-topic=/waku/2/rs/16/32 \
   --start-time=1722424194452637398 \
   --end-time=1722425909559247654 \
   --content-topic=/waku/1/0xa4d9d91b/rfc26 \
   --content-topic=/waku/1/0x42c4e775/rfc26 \
   --storenode=/dns4/store-01.do-ams3.status.prod.status.im/tcp/30303/p2p/16Uiu2HAmAUdrQ3uwzuE4Gy4D56hX6uLKEeerJAnhKEHZ3DxF1EfT \
   --use-legacy=false    <-------- change to true to compare

Legacy:

Page: 1, Record from 1 to 20 (2.401µs)
MessageHash                                                         PubsubTopic       Content Topic             Timestamp                                                     
0xb04336e653d453905f639d7e09b2241c4fb08984243cc0ee9bc1bef364bd90fa  /waku/2/rs/16/32  /waku/1/0x42c4e775/rfc26  1722425876782613943  2024-07-31 11:37:56.782613943 +0000 UTC  
...

Store-v3:

Page: 1, Record from 1 to 20 (1.704460297s)
MessageHash                                                         Content Topic             Timestamp                                                     
0xb04336e653d453905f639d7e09b2241c4fb08984243cc0ee9bc1bef364bd90fa  /waku/1/0x42c4e775/rfc26  1722425876782613943  2024-07-31 11:37:56.782613943 +0000 UTC  
...

storev3 consistently takes more than 600ms to return a page while legacy consistently is < 300ms for the same query

Queries used:

// StoreV3
{
  "request_id": "5a830c991c2adc38ab5a5869905a6b5c0066ff946a6ac98b2503addd0c529937",
  "include_data": true,
  "pubsub_topic": "/waku/2/rs/16/32",
  "content_topics": ["/waku/1/0xa4d9d91b/rfc26", "/waku/1/0x42c4e775/rfc26"],
  "time_start": 1722424194452637398,
  "time_end": 1722425909559247654,
  "pagination_limit": 20
}

// Legacy
{
    "request_id": "a84e648c154c68dae755cd6f1a302e76598d7d0b7b76b040ad7ea7c727baa704",
    "query": {
      "pubsub_topic": "/waku/2/rs/16/32",
      "content_filters": [
        { "content_topic": "/waku/1/0xa4d9d91b/rfc26" },
        { "content_topic": "/waku/1/0x42c4e775/rfc26" }
      ],
      "paging_info": { "page_size": 20 },
      "start_time": 1722424194452637398,
      "end_time": 1722425909559247654
    }
}