Substrate extrinsics and events filtering

[stwiname]

Polkadot Dictionary Pallet

Background

Finding blocks that contain relevant data currently requires looking at each block one by one. To improve this SubQuery has created its own dictionaries that have an index of the types of extrinsics and events are in each block. This has a few downsides and there is much room for improvement through a more integrated solution

Implementation

The aim is to introduce a new FRAME Pallet that creates indexes for extrinsics and events in each block as well as offering an RPC interface to utilise the indexes.

The pallet should index all historical blocks as well as new blocks and deal with finalization.

• The index data should be persisted in storage.
• The index should include module and call, success for extrinsics and module and event for events. It doesn't need to include any arguments
• It should use bloom filter, though a new filter looks promising, it will be nice if some comparison can be done there. Cuckoo Filters. Rust Impl, C++ Impl

It should also implement the RPC interface as described below:

RPC Interface

These requests and responses are intended to be used with JSON-RPC. This way they can be integrated into existing RPC services.

Methods

subql_filterBlocksCapabilities

Params
None

Result
CAPABILITY:

type Capability = {
  availableBlocks: [startHeight: number, endHeight: number];
  // Describes what is available to be filtered
  filters: Record<
    string, // Entity name
    string[]
  >;
  // Describes the possible response data fields that can be returned. These are defined by us. e.g 'basic', 'complete', 'trace'
  supportedResponses: string[];
  genesisHash: string; // The chains first block hash, used to identify the network
}

V1 based dictionaries could provide ‘basic’ responses, while V2 could provide complete + more responses. Archive nodes should provide 'complete' responses (extrinsics and events) while a full node should be able to provide basic responses (headers)

Note: leave possibility for more options provided by the service. e.g. max block range, complexity

Example Response

{
  availableBlocks: [1000000, 54321000],
  filters: {
    extrinsics: ['module','call'],
    events: ['module', 'event'],
  },
  supportedResponses: ['basic', 'complete'],
  genesisHash: '0x05a56E2D52c817161883f50c441c3228CFe54d9f'
}

subql_filterBlocks

Get the blocks that match the filter

Params

1. NUMBER - The query start block
2. NUMBER - The query end block
3. NUMBER - Limit to the number of blocks in the return
4. BLOCK_FILTER - This determines the blocks returned as well as the content within the blocks. TODO define filter matching behaviour including null, *, case sensitivity etc

type EntityFilter = Record<
  string, // Entity field. E.g. module, event
  any[]
>; // AND filtering on the entity filter, meaning that all fields much have a matching condition

// Using a record ensures uniqueness of entity
type BlockFilter = Record<
  string, // Entity name. E.g. Call, Event
  EntityFilter[], // Filters for this event type, OR filtering
>; // OR filtering on any block amongst all entity filters

5. FIELD_SELECTOR (Optional) - Specifies the fields returned in the block response. If this is undefined then just the header will be returned.

type FieldSelector = Record<
  string, // The entity name
  boolean
>;

Result

BLOCK_RESULT - Contains information about the blocks that match the filter as well as the block range searched.

If a FIELD_SELECTOR is provided then all entities requested will be returned for matching blocks.

// Use a named tuple here to reduce size of data
  type Block = Header & any; The block header and any other fields defined by the FIELD_SELECTOR

  type BlockResult = {
    blocks: Block[]; // An array of Blocks, this could be empty
    blockRange: [
      start: number; // The block height the query started at, inclusive
      end: number; // The last block the filter was applied to, inclusive
    ];
    genesisHash: string; // The chains first block hash, used to validate the correct chain
  };

Example

Request:

[
  1,
  10000,
  20,
  {
    events: [
      { method: 12, event: 3}, // balances.Deposit
      { method: 12, event: 1}, // balances.Transfer
    ],
    extrinsics: [
      { method: 12, call: 5 } // balances.Deposit
    ],
  },
  {
    blockHeader: true, // always true
    extrinsics: true,
    events: true,
  }
]

Response:

{
  blocks: [
    {
      header: { height: 1, hash: '0xABC', ...},
      extrinsics: /* encoded extrinsics */,
      events: /* encoded events */
    },
    ...more blocks
  ],
  // [start, end], this indicates the height we can proceed forward, 
  // it isn't neccessarily overlap with the requested block range
  blockRange: [1, 10000],
  genesisHash: '0x05a56E2D52c817161883f50c441c3228CFe54d9f'
}

Error Handling

Filter validation should be very strict. If an entity or field doesn’t exist then an error should be thrown. Same applies for unknown filter operators.

If a request is made for a block range that starts before the service has blocks then it should throw an error.

If the service only provides basic supported responses and FIELD_SELECTOR is provided then an error should be thrown.

References

paritytech/polkadot-sdk#1532
paritytech/polkadot-sdk#278