RPC Method to get a stream of Nullifiers for Block Range

[pacu]

What is your feature request?
I'd like to be able to get the nullifiers first so I can know which funds are spendable for a given range

How would this feature help you?
It will make syncing faster and more resource efficient

fileds needed

Blockheight
for everyTx:

• txid
• sapling nullifiers
• nullifier orchard actions

[LarryRuane]

Let me try to make it more concrete: The argument would be a block range (starting and ending height), and the return would be a stream of per-transaction structs, sorted by block height, each containing:

• block height
• txid
• list of ShieldedSpend (sapling) nullifiers (bytes? or hex strings?)
• list of Action (Orchard) nullifiers (bytes? or hex strings?)

Or, we could do a stream of per-block objects, each containing

• block height
• list of {
  • txid
  • list of ShieldedSpend nullifiers
  • list of Action nullifiers
• }

If it's possible for a single block to have a large number of transactions, then the first alternative might be better, because it's more "streamy" -- the second alternative would also be streaming, but each item (block) would be a fairly big non-streamed thing. The advantage of the second alternative is block height wouldn't be sent multiple times. But height is pretty small, 8 bytes (it really could be 4 but we've always encoded height as an 8-byte integer, but that's still pretty small).

But whichever is easier for you to parse should probably determine it (I think it's about the same complexity and performance for lightwalletd either way). Or maybe there's some other interface that would be better than either of these. For example, a stream of:

• block height
• txid
• type (sapling or action)
• nullifier (either a list or just one)

But then we'd be sending both a particular block height and txid multiple times, requiring more bandwidth.

[pacu]

Yes, that sounds about right Larry!

[LarryRuane]

Yes, that sounds about right Larry!

I gave some alternatives, do you have a preference?

[str4d]

My assumption is that it would be the second case, i.e. take the CompactBlock format and leave some data out. I also wondered if this could be done as a modification to the existing getblock stream by taking an optional filter argument. It might not even be necessary to send txids in this data stream, as long as the index of the transaction within the block is known (which is implicit in the second / getblock format).

[pacu]

Yes, that sounds about right Larry!

I gave some alternatives, do you have a preference?

Oh sorry I misunderstood.

I think that me must favor the most "streamy" as we can since is more gentle on clients' resources and probably on the server as well .

[str4d]

For DAGSync usage, we need the second case, because what we need to construct from this stream is the map nullifer -> (block_reference, tx_index_in_block).

• We want this to be nullifer -> (block_reference, tx_index_in_block) and not nullifer -> txid because the outputs we subsequently fetch for trial decryption will be fetch at a per-block level, not a per-transaction level, so we need to know the block that the transaction is in. We will need to learn the txid eventually, but we can reduce overall bandwidth by only downloading txids during the compact output stage (which can be bounded in height) rather than the nullifier stage (which inherently needs to fetch every block eventually).
• block_reference needs to include at least the height, and ideally the block hash as well (because reorgs can occur in between calls to this API, so we need to ensure that the wallet can detect this and trigger a rewind).
  • If this still used too much bandwidth, we could maybe prune this to only include block hashes for blocks between chain_tip_height - 100 and the chain tip, as reorgs are highly unlikely to occur outside that range.
• tx_index_in_block is conveyed as the index field of CompactTx, although we could potentially convey it indirectly by providing empty CompactTxs in the spots where there are transparent-only transactions. We could measure how large the two options are on average; either approach would work.

So what I would propose is that we take existing CompactBlocks that lightwalletd can already be caching, and strip out the data we don't need. This is fine because proto3 allows every field to be omitted. Our fields are all inherently type singluar, so during parsing, the omitted fields would be "parsed" as their default values, but we would know to ignore them.

So we just have:

CompactBlock {
    protoVersion, // We may want to bump this to indicate it should be interpreted as nullifiers-only, not a full CompactBlock.
    height,
    hash,
    vtx: [CompactTx {
        index,
        spends: [CompactSaplingSpend {
            nf,
        }],
        actions: [CompactOrchardAction {
            nullifier,
        }],
    }],
}

[LarryRuane]

Thanks, @strd!

[LarryRuane]

Sorry for the delay; I've got most of this working, but haven't made a PR yet. Can you verify that this is what you expect? To test, I started a local mainnet lightwalletd and ran:

$ grpcurl -plaintext -d '{"height":2034294}' localhost:9067 cash.z.wallet.sdk.rpc.CompactTxStreamer/GetBlock

Without my changes (current master), the beginning of the output is:

{
  "height": "2034294",
  "hash": "qiCzrUzmutvuvkJzpX3k/mKiNZ+CHbFbZ5eKAAAAAAA=",
  "prevHash": "Wvo9U9ggLn8RA115u+FmXYV/L12XsFeBkz0pAAAAAAA=",
  "time": 1680188063,
  "vtx": [
    {
      "index": "1",
      "hash": "Fte7wSWshqy9ZUTy4pYRkaUE012GeZgqT0z5UpISdG0=",
      "actions": [
        {
          "nullifier": "2EZAprlW5C826U5pnLIoT0NIKei9EcZCCmWzqf5o/iY=",
          "cmx": "a8UwbT0i9DWgGDFP6E7WBk/EfLrYipNfssLvLsrR8DM=",
          "ephemeralKey": "p5XsVt2yYoBlNPt6kAXlT4cY84ir9iCaXbBvLcQNEpw=",
          "ciphertext": "pgESOG9OQMMvTvk1WAWpOKPYQEz42nLn9bJFOVmZxFRVMrDeMTv59yPMGu7nQylxUdye6A=="
        },
        {
          "nullifier": "WSYkayvIq+SjpFgq7ebpHtr0KhXBL7OXPs9gQBBioCs=",
          "cmx": "nGU+BRVhh6QdyjzBaIAquMUSc024IXYpxmM2skMLAjY=",
          "ephemeralKey": "+5ayDlY0CUkYdFbG0W36eGJ4x7Vh/aulNKytK3O2SxE=",
          "ciphertext": "G04mHUzCa4XQLoBLJD30Ce4ydjKAOC2lJiPE9kfkeXiqOK1tlsg/OH7UBiPd87sxZAbLzg=="
        },
(... many more actions ...)

With my changes, the first part is the same but then:

      "actions": [
        {
          "nullifier": "2EZAprlW5C826U5pnLIoT0NIKei9EcZCCmWzqf5o/iY="
        },
        {
          "nullifier": "WSYkayvIq+SjpFgq7ebpHtr0KhXBL7OXPs9gQBBioCs="
        },

Is that what we want?

It turns out that for Sapling spends, we already return only the nullifier, so I don't think any change is needed there.

I haven't implemented the changes to the gRPC interface yet. Instead of a new flag (to indicate that lightwalletd should return this new compact-compact block), it looks like the best (and arguably only) way is to have two new gRPCs that correspond to the existing GetBlock and GetBlockRange gRPCs. Is that okay? There are probably multiple ways for your code to query the server to find out if it supports these RPCs. If nothing else, you can do this and check if the new ones are present:

$ grpcurl -plaintext localhost:9067 list cash.z.wallet.sdk.rpc.CompactTxStreamer
cash.z.wallet.sdk.rpc.CompactTxStreamer.GetAddressUtxos
cash.z.wallet.sdk.rpc.CompactTxStreamer.GetAddressUtxosStream
cash.z.wallet.sdk.rpc.CompactTxStreamer.GetBlock
cash.z.wallet.sdk.rpc.CompactTxStreamer.GetBlockRange
cash.z.wallet.sdk.rpc.CompactTxStreamer.GetLatestBlock
(...)

Do you have any suggestion for the names of these new methods? Maybe

• GetBlockNullifiers
• GetBlockRangeNullifiers

They would take the same arguments as the corresponding existing methods. Is that okay?

[pacu]

I think the info needed is there.

I'd ask @str4d if we want to model it in a fashion that it can be converted to a map in a more straight forward way or if this is OK.