Non-deterministic output from get table with large limits

cleos get table outputs different number of rows when a large limit is applied, on that case the symptoms started around 1666 rows and at very large values it doesn't output the full data nor give any warning.

Strange behaviour:

Correct behaviour:

cleos fetching a high row:

cleos version: 1a27ee3a
nodeos version: 438824506
v1.0.1

We are experiencing this as well, even when using limits of 300 or 400.

At the end of the received json is the "more":true when not everything was retrieved. How to we enforce nodeos to get the whole table that we requested?

This is a big issue with listing producers, as it would only show ones high in alphabetical order.

Tokenika has same problem. It makes it hard to follow changes via API. 👎

eos/plugins/chain_plugin/include/eosio/chain_plugin/chain_plugin.hpp

Line 312 in c55061b

    
           auto end = fc::time_point::now() + fc::microseconds(1000 * 10); /// 10ms max time

This is the problem: there is an un-configurable 10ms timeout.

This should be configurable!

It also seems strange that it's taking more than 10ms to get the list of producers.

We have hotfixed this on our https://eos.greymass.com API node, but a better solution is needed.

If you increase the timeout too much you risk your API node being unresponsive. So there needs to be a sensible time limit. And yes perhaps that limit should be configurable in the config.ini.

But fundamentally, consumers of the RPC API will need to expect that some requests will take too long (or be too big) to fulfill all at once. This is the reason for the more boolean. The consumer can know that there are more rows in the table, so it knows to request more of them if it wants them all.

The way to do this with cleos is to use the -L option.

For example:
The initial request could be something like the following (but in practice with a larger limit than 5):

$ ./cleos get table -l 5 eosio.unregd eosio.unregd addresses 
{
  "rows": [{
      "id": 0,
      "ethereum_address": "0x89a61a6d6eb95f56da5bb2f5e879a569d85967ef",
      "balance": "1135.8899 EOS"
    },{
      "id": 1,
      "ethereum_address": "0x8d12a197cb00d4747a1fe03395095ce2a5cc6819",
      "balance": "74483.0001 EOS"
    },{
      "id": 2,
      "ethereum_address": "0x74c2320e0d2ace5d958f22cd56eabc4adb99e340",
      "balance": "113.0000 EOS"
    },{
      "id": 3,
      "ethereum_address": "0xd816fca4210adae28630c6bec091e7c370d1c614",
      "balance": "1.9000 EOS"
    },{
      "id": 4,
      "ethereum_address": "0xa083804609f5aed89d45c928940781e6179023b0",
      "balance": "1.0000 EOS"
    }
  ],
  "more": true
}

Then the consumer of this data could see that there is more data available, and the row with the largest primary key is 4. So, it makes another request:

$ ./cleos -L 4 -l 5  eosio.unregd eosio.unregd addresses 
{
  "rows": [{
      "id": 4,
      "ethereum_address": "0xa083804609f5aed89d45c928940781e6179023b0",
      "balance": "1.0000 EOS"
    },{
      "id": 5,
      "ethereum_address": "0xdb9f899415661b78105177d0308a0df3bf2a2f60",
      "balance": "6.0000 EOS"
    },{
      "id": 6,
      "ethereum_address": "0xc7f1e4801cb955894f77031205a8fa27b40c0bf7",
      "balance": "6.4035 EOS"
    },{
      "id": 7,
      "ethereum_address": "0xdbe8b6cf5ad2242a0c086d827abb00068804b265",
      "balance": "1.0000 EOS"
    },{
      "id": 8,
      "ethereum_address": "0x217a6df28495ee7250ce05c6150055c3e711666f",
      "balance": "303.7265 EOS"
    }
  ],
  "more": true
}

It can skip over the row with primary key 4 since it has seen it already, but it now has four additional rows it never saw before. Since there are more rows available, it can continue to retrieve more:

$ ./cleos get table -L 8 -l 5  eosio.unregd eosio.unregd addresses 
{
  "rows": [{
      "id": 8,
      "ethereum_address": "0x217a6df28495ee7250ce05c6150055c3e711666f",
      "balance": "303.7265 EOS"
    },{
      "id": 9,
      "ethereum_address": "0x1aea842285337bb4840ecf03ea80b88128f27817",
      "balance": "30.6794 EOS"
    },{
      "id": 10,
      "ethereum_address": "0x984d6667f253c22381eebfaa11fe3351edbae291",
      "balance": "80.0436 EOS"
    },{
      "id": 11,
      "ethereum_address": "0x216a3eae236f2f3f779c9901835f87c275c9e172",
      "balance": "1.0000 EOS"
    },{
      "id": 12,
      "ethereum_address": "0x1de274c5399b18ae2289097f60deea6bf23edcef",
      "balance": "27.1780 EOS"
    }
  ],
  "more": true
}

And so on.

That -L option stands for lower bound, and you can provide it in the HTTP request using the lower_bound field. For example, the previous cleos command sent the following request (which you can easily discover using the --print-request option):

POST /v1/chain/get_table_rows HTTP/1.0
Host: localhost
content-length: 171
Accept: */*
Connection: close

{
  "json": true,
  "code": "eosio.unregd",
  "scope": "eosio.unregd",
  "table": "addresses",
  "table_key": "",
  "lower_bound": "8",
  "upper_bound": "",
  "limit": 5
}

The problem of course is consistency between the requests. Perhaps that is not a serious issue for your particular application (as it may not be in many cases where pagination is used).

If consistency is really a major concern for your particular use case, then what I think you would really want is an ability to pause the node at the state at the end of some block number so that you can make all the read requests you want before resuming. Obviously this mode of operation is unacceptable for a public API node.

Feel free to make a feature request for such a pausing ability. Or a better yet a PR 😊.

For those reading that later, you can use https://docs.dfuse.io/#rest-get-v0-state-table to get consistent snapshots of tables at any block height.