Improve performance of rosetta-cli check:data testing

Question

Improve performance of rosetta-cli check:data testing

Closed this issue 3 years ago · 45 comments

jimni1222 commented 3 years ago

Improve performance of rosetta-cli check:data testing.

To test rosetta-klaytn with real data in Cypress and Baobab, we need to improve performance to reduce testing time.

Substask of #8

Answer 1 · 2022-05-25T07:22:02.000Z

Instance type:
c5.4xlarge

Configuration
Base: https://github.com/coinbase/rosetta-cli/blob/master/examples/configuration/default.json

What we changed here

{
  "network": {
    "blockchain": "Klaytn",
    "network": "Testnet"
  },
  "online_url": "http://localhost:9090",
  "data_directory": "cli-data-testnet",
  "http_timeout": 120,
  "max_retries": 15,
  "data": {
    "bootstrap_balances": "bootstrap_balances.json",
    "end_conditions": {
      "index": 20541599
    }
  }
}

start_index: 20491599
will sync 50,000 blocks

[UPDATED]
Server went down with log below.

2022/05/25 09:39:10 check:data status server shutting down
badger 2022/05/25 09:39:10 INFO: Storing value log head: {Fid:3148 Len:37 Offset:36667821}
Error: unable to lookup live balance: request failed: /account/balance {Code:2 Message:klaytn client error Description:<nil> Retriable:false Details:map[context:Post "http://15.164.192.30:8551": dial tcp 15.164.192.30:8551: connect: connection refused]}: unable to get live balance for {"address":"0xE377D25Dc75Bc6516d9627B2340A9735607c8DFe"} {"symbol":"KLAY","decimals":18} at 20491882

I'm currently investigating why this "connection refused" happen and try to resolve that problem.
The possible reason what I think is

Request from rosetta-cli exceeded concurrency which EN can handle
- If this is a reason, we can solve this problem by increasing concurrency value of EN.
Too many open files

Answer 2 · 2022-05-30T07:31:36.000Z

I adjusted kernel parameter like below for all Archive ENs.

Cypress Archive EN(=52.78.55.68)
Baboab Archive EN(=15.164.192.30)
Baobab Archive EN2(=3.39.138.249)

`oepn files`	`net/core/somaxconn`	`net/ipv4/tcp_max_syn_backlog`
1024 -> 65535	128 -> 9182	2048 -> 65536

Sync progress

Time	Baobab / bps	Cypress / bps	Kernel Parameters	Elapsed
05-30 16:30	27039671	21677030	kernelParameters
05-31 08:23	32220090 / 90.6	22752574 / 18.8	kernelParameters	57180
06-02 14:52	48860192 / 84.8	26020710 / 16.6	kernelParameters	196140

Applied retry

Remaining days based on current speed

Cypress: 46 days
Baobab: 6 days

@jimni1222

Answer 3 · 2022-06-02T13:01:39.000Z

2022.06.03 Test Configuration Update

Cypress
- Cypress Archive EN 1 (from 0 block)
- Cypress Archive EN 2 (from 60,000,000 block) - Not fully synced, so test speed is slow
- Cypress Archive EN 3 (from 40,000,000 block) - Not fully synced, so test speed is slow
- Cypress Archive EN 4 (from 80,000,000 block) - Not fully synced, so test speed is slow
- Cypress Archive EN 5 (from 70,000,000 block) - Not fully synced, so test speed is slow
Baobab
- Baobab Archive EN 1 (from 70,000,000 block)
- Baobab Archive EN 2 (from 0 block)

From the end of 70,000,000 block to start of 80,000,000 block, there are really many transactions in cypress block.
I thought we should separate section between 60,000,000 ~ 80,000.000, so i ran Cypress EN5 for testing from 70,000,000 block.

https://www.klaytnfinder.io/block/78170250
https://www.klaytnfinder.io/block/79611941
https://www.klaytnfinder.io/block/79711208
https://www.klaytnfinder.io/block/80044270

@aeharvlee

Answer 4 · 2022-06-03T04:52:47.000Z

If you define the bootstrap_balances option as a json file, it is used to set the initial balance when performing the test, so if the check:data test is performed from the middle, reconcile for the account defined in bootstrap_balances fails.

Therefore, this option should be omitted if you are running your tests in the middle.

Network	Block Number	Status	EN	22.06.06 23:39 Block Number
Cypress	0~40,000,000	Testing	EN1	32,327,171
Cypress	40,000,000~60,000,000	Testing	EN3	43,238,130
Cypress	60,000,000~70,000,000	Testing	EN2	62,466,305
Cypress	70,000,000~80,000,000	Testing	EN5	72,567,267
Cypress	80,000,000~	Test failed with `80094047`	EN4	-

Network	Block Number	Status	EN	22.06.06 23:40 Block Number
Baobab	0~70,000,000	Done	-	-
Baobab	70,000,000~88,000,000	Testing	EN1	-
Baobab	88,000,000~	Testing	EN2	88,335,977

Network	Range	Status	EN	Data-Dir	2022. 6. 13. Block Number
Cypress	0~40,000,000	Done	-	-	-
Cypress	40,000,000~53,000,000	Testing	EN3	{dir-prefix}-40000000	48,796,786
Cypress	53,000,000~60,000,000	Testing	EN7	{dir-prefix}-53000000	56,282,984
Cypress	60,000,000~70,000,000	Testing	EN2	{dir-prefix}-60000000	67,273,038
Cypress	70,000,000~78,000,000	Testing	EN5	{dir-prefix}-70000000	75,684,127
Cypress	78,000,000~80,000,000	Testing	EN6	{dir-prefix}-78000000	78,093,072
Cypress	80,000,000~83,000,000	Testing	EN4	{dir-prefix}-80000000	81,142,990
Cypress	83,000,000~86,000,000	Testing	EN1	{dir-prefix}-83000000	83,122,388
Cypress	86,000,000~93,192,251	Done	-	-	-