[BUG]: Unexpected stop during sync

Question

[BUG]: Unexpected stop during sync

tenonedesign opened this issue 2 years ago · 6 comments

Is there an existing issue for this?

I have searched the existing issues

Current behavior

Noticed block_store stopped running during a sync. I restarted it after 8 minutes and the sync completed from there. Block height was at or near 1776053.

I don’t have a combined log. This is the block_store log.

Prior to crash___________________

2022-10-26 01:27:03.107615 (block_store.Koinos) [koinos-block-store/main.go:205] <info>: Recently added 71 block(s)
2022-10-26 01:27:57.057293 (block_store.Koinos) [koinos-block-store/main.go:171] <info>: Sync block progress - Height: 1774000, ID: 0x1220b9445f1e935746d12671bbe1c58988eb02d4d8d25900dcc5e55f7cc55286df6f
2022-10-26 01:28:03.067430 (block_store.Koinos) [koinos-block-store/main.go:205] <info>: Recently added 969 block(s)
2022-10-26 01:28:23.469535 (block_store.Koinos) [koinos-block-store/main.go:171] <info>: Sync block progress - Height: 1775000, ID: 0x12206e4c2c5c3740262d26e41b3a03784f29055df072691ed39c4a022b48b25f0ce6
2022-10-26 01:29:03.026148 (block_store.Koinos) [koinos-block-store/main.go:205] <info>: Recently added 1265 block(s)
2022-10-26 01:30:02.985950 (block_store.Koinos) [koinos-block-store/main.go:205] <info>: Recently added 82 block(s)
2022-10-26 01:31:02.946249 (block_store.Koinos) [koinos-block-store/main.go:205] <info>: Recently added 98 block(s)
2022-10-26 01:32:02.905836 (block_store.Koinos) [koinos-block-store/main.go:205] <info>: Recently added 72 block(s)
2022-10-26 01:33:02.865454 (block_store.Koinos) [koinos-block-store/main.go:205] <info>: Recently added 67 block(s)
2022-10-26 01:34:02.824835 (block_store.Koinos) [koinos-block-store/main.go:205] <info>: Recently added 74 block(s)
2022-10-26 01:34:52.558505 (block_store.Koinos) [koinos-block-store/main.go:171] <info>: Sync block progress - Height: 1776000, ID: 0x1220e01cb68e3cba4adb2bf23147245c11240bb09dbf2a8d54a98877073c40e94691
2022-10-26 01:35:02.785016 (block_store.Koinos) [koinos-block-store/main.go:205] <info>: Recently added 76 block(s)
2022-10-26 01:36:02.745137 (block_store.Koinos) [koinos-block-store/main.go:205] <info>: Recently added 90 block(s)
2022-10-26 01:36:13.227501 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] <info>: [0] [E] LOG Compact 0->5 (5, 34 -> 41 tables with 5 splits). [04256 04257 04258 04259 04260 . 04212 04216 04221 04227 04230 04237 04240 04243 04214 04217 04224 04228 04233 04238 04244 04213 04219 04222 04226 04232 04236 04241 04215 04220 04223 04229 04235 04246 04247 04211 04218 04225 04231 04234 .] -> [04261 04265 04267 04276 04281 04285 04290 04294 04264 04268 04275 04279 04284 04288 04295 04297 04263 04269 04273 04278 04282 04287 04292 04296 04298 04262 04271 04274 04280 04283 04289 04299 04300 04301 04266 04270 04272 04277 04286 04291 04293 .], took 4.808s
2022-10-26 01:36:24.304205 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] <info>: [0] [E] LOG Compact 5->6 (1, 4 -> 4 tables with 2 splits). [04297 . 04021 04022 03591 03592 .] -> [04305 04310 04312 04302 .], took 11.074s
2022-10-26 01:36:26.370253 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] <info>: [3] [E] LOG Compact 5->6 (1, 6 -> 6 tables with 2 splits). [04283 . 04011 04014 03587 03432 03434 03990 .] -> [04304 04311 04313 04309 04315 04319 .], took 13.132s
2022-10-26 01:36:26.485731 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] <info>: [2] [E] LOG Compact 5->6 (1, 3 -> 3 tables with 1 splits). [04296 . 03570 03574 03228 .] -> [04307 04314 04316 .], took 13.207s

After restart___________________

2022-10-26 01:44:06.974497 (block_store.Koinos) [koinos-block-store/main.go:102] <info>: Opening database at /koinos/block_store/db
2022-10-26 01:44:07.481414 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] <info>: All 251 tables opened in 361ms
2022-10-26 01:44:07.483659 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] <info>: Discard stats nextEmptySlot: 0
2022-10-26 01:44:07.484769 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] <info>: Set nextTxnTs to 5457665
2022-10-26 01:44:08.100392 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] <info>: Deleting empty file: /koinos/block_store/db/000097.vlog
2022-10-26 01:44:08.153381 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221024231917-d90ee8d48910/request_handler.go:100] <info>: Connecting request handler to AMQP server amqp://guest:guest@amqp:5672/
2022-10-26 01:44:08.156195 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221024231917-d90ee8d48910/connection.go:85] <info>: Dialing AMQP server amqp://guest:guest@amqp:5672/
2022-10-26 01:44:08.191778 (block_store.Koinos) [koinos-mq-golang@v0.0.0-20221024231917-d90ee8d48910/request_handler.go:132] <info>: Request handler connected
2022-10-26 01:44:11.960978 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] <info>: [2] [E] LOG Compact 5->6 (1, 7 -> 7 tables with 3 splits). [04275 . 03821 03822 03811 03813 03806 03809 03362 .] -> [04324 04328 04329 04322 04331 04341 04321 .], took 4.405s
2022-10-26 01:44:12.461733 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] <info>: [3] [E] LOG Compact 5->6 (1, 7 -> 9 tables with 3 splits). [04272 . 03837 03844 03845 03827 03839 03846 03510 .] -> [04325 04335 04339 04323 04336 04343 04344 04326 04327 .], took 4.456s
2022-10-26 01:44:12.488375 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] <info>: [0] [E] LOG Compact 5->6 (1, 4 -> 5 tables with 2 splits). [04285 . 03808 03812 03814 03803 .] -> [04338 04345 04347 04332 04333 .], took 3.951s
2022-10-26 01:44:12.496310 (block_store.Koinos) [v3@v3.2103.2/logger.go:46] <info>: [1] [E] LOG Compact 5->6 (1, 6 -> 6 tables with 2 splits). [04289 . 04315 04319 04001 03992 03996 03679 .] -> [04334 04342 04346 04330 04337 04340 .], took 4.21s
2022-10-26 01:45:08.112563 (block_store.Koinos) [koinos-block-store/main.go:205] <info>: Recently added 14 block(s)
2022-10-26 01:46:08.073429 (block_store.Koinos) [koinos-block-store/main.go:205] <info>: Recently added 18 block(s)
2022-10-26 01:47:08.036374 (block_store.Koinos) [koinos-block-store/main.go:205] <info>: Recently added 42 block(s)

Expected behavior

No response

Steps to reproduce

No response

Environment

- OS: MacOS 12.6, 2019 MBP
- Docker desktop 4.8.1

All images at latest as of 2022-10-26 00:48:00.732550Z (about 8:48pm eastern)

Anything else?

No response

Answer 1 · 2022-10-26T02:28:52.000Z

Nothing in these logs look wrong. There is not an error message. The docker container was stopped?

Answer 2 · 2022-10-26T02:33:20.000Z

Yes. No indication of why anywhere that I could see.

Answer 3 · 2022-10-26T02:50:29.000Z

As an attempt to keep this from being the least useful bug report in history, I'm pasting in all the context I could find from other logs.

p2p_____
2022-10-26 01:34:59.163770 (jsonrpc.Koinos) [koinos-jsonrpc/main.go:287] : Recently handled 24 request(s)
2022-10-26 01:35:59.122857 (jsonrpc.Koinos) [koinos-jsonrpc/main.go:287] : Recently handled 24 request(s)
2022-10-26 01:36:59.084754 (jsonrpc.Koinos) [koinos-jsonrpc/main.go:287] : Recently handled 24 request(s)
2022-10-26 01:37:59.044444 (jsonrpc.Koinos) [koinos-jsonrpc/main.go:287] : Recently handled 24 request(s)
2022-10-26 01:38:59.005274 (jsonrpc.Koinos) [koinos-jsonrpc/main.go:287] : Recently handled 9 request(s)
2022-10-26 01:39:58.960679 (jsonrpc.Koinos) [koinos-jsonrpc/main.go:287] : Recently handled 7 request(s)
2022-10-26 01:40:58.918587 (jsonrpc.Koinos) [koinos-jsonrpc/main.go:287] : Recently handled 10 request(s)
2022-10-26 01:41:58.878377 (jsonrpc.Koinos) [koinos-jsonrpc/main.go:287] : Recently handled 24 request(s)
2022-10-26 01:42:58.838571 (jsonrpc.Koinos) [koinos-jsonrpc/main.go:287] : Recently handled 24 request(s)
2022-10-26 01:43:58.799599 (jsonrpc.Koinos) [koinos-jsonrpc/main.go:287] : Recently handled 24 request(s)
2022-10-26 01:44:58.760623 (jsonrpc.Koinos) [koinos-jsonrpc/main.go:287] : Recently handled 24 request(s)

transaction_store_____
2022-10-26 01:34:06.081132 (transaction_store.Koinos) [koinos-transaction-store/main.go:193] : Recently added 30212 transaction(s)
2022-10-26 01:34:52.559125 (transaction_store.Koinos) [koinos-transaction-store/main.go:164] : Sync block progress - Height: 1776000, ID: 0x1220e01cb68e3cba4adb2bf23147245c11240bb09dbf2a8d54a98877073c40e94691
2022-10-26 01:35:06.040694 (transaction_store.Koinos) [koinos-transaction-store/main.go:193] : Recently added 32920 transaction(s)
2022-10-26 01:36:05.999614 (transaction_store.Koinos) [koinos-transaction-store/main.go:193] : Recently added 32918 transaction(s)
2022-10-26 01:37:05.966083 (transaction_store.Koinos) [koinos-transaction-store/main.go:193] : Recently added 2703 transaction(s)
2022-10-26 01:45:05.639239 (transaction_store.Koinos) [koinos-transaction-store/main.go:193] : Recently added 5405 transaction(s)

chain_____
2022-10-26 01:28:23.464623 (chain.Koinos) [controller.cpp:371] : Sync progress - Height: 1775000, ID: 0x12206e4c2c5c3740262d26e41b3a03784f29055df072691ed39c4a022b48b25f0ce6 (02h, 30m, 23s block time remaining)
2022-10-26 01:34:52.544456 (chain.Koinos) [controller.cpp:371] : Sync progress - Height: 1776000, ID: 0x1220e01cb68e3cba4adb2bf23147245c11240bb09dbf2a8d54a98877073c40e94691 (01h, 30m, 48s block time remaining)
2022-10-26 01:36:16.872900 (chain.Koinos) [controller.cpp:472] : Block application failed - Height: 1776113 ID: 0x122004ed6ad4876561e1b4088bf723a508c3dea91a1c8c9f38d151e1fbf21935b253, with reason: request timeout: 9nR3RdfGk5AjAJm4jQIAkfzu41z3OYw9
2022-10-26 01:36:17.418210 (chain.Koinos) [client.cpp:362] : Client could not find correlation ID in request set: 9nR3RdfGk5AjAJm4jQIAkfzu41z3OYw9
2022-10-26 01:45:22.974685 (chain.Koinos) [controller.cpp:472] : Block application failed - Height: 1776139 ID: 0x12205e63aef4a3f4585b0979a3edeea950b5b1b6de168f44fffddbc34be19e3728f0, with reason: request timeout: orQsgL70lDKSju1rSeRZARMOUbkRB06G
2022-10-26 01:45:40.063918 (chain.Koinos) [client.cpp:362] : Client could not find correlation ID in request set: orQsgL70lDKSju1rSeRZARMOUbkRB06G
2022-10-26 01:45:46.020729 (chain.Koinos) [controller.cpp:472] : Block application failed - Height: 1776142 ID: 0x1220f1f7cfc916bc6bc29576b5dd269228381a994d6d3fdbded272f4bf125a11855b, with reason: request timeout: rOpmW8SDWlO4lm3POmYMp1OrbpGcAdK4

p2p_____
2022-10-26 01:36:23.845020 (p2p.Koinos) [node/node.go:371] : - /ip4/192.168.0.8/tcp/8888/p2p/QmYhPfRykBVTL2xnHH7igXuqVis4LRsxgbh4AmV2Hg1hpU
2022-10-26 01:36:23.846685 (p2p.Koinos) [node/node.go:372] : Connected peers:
2022-10-26 01:36:23.847832 (p2p.Koinos) [node/node.go:374] : - /ip4/139.144.17.130/tcp/8888/p2p/QmV7q74bXxbg4yGY1kPgj9LsJPjgXqwc9SPtdTiSS67tvk
2022-10-26 01:36:23.848246 (p2p.Koinos) [node/node.go:374] : - /ip4/64.62.202.13/tcp/8888/p2p/QmTmTvVcSQd73qXt6xGTrHwaEwjKZgaQ5ZkxYJzTkFfimH
2022-10-26 01:36:23.848693 (p2p.Koinos) [node/node.go:374] : - /ip4/185.130.47.36/tcp/8888/p2p/QmdrdcEnYDLgJUZzrhYNhYvEifBhiiTn7bndmnJUTma7dB
2022-10-26 01:36:23.849133 (p2p.Koinos) [node/node.go:374] : - /ip4/51.15.5.91/tcp/8888/p2p/QmUwUmKLiQzMDg3xWkpfUnsbvBsrXK1uVqpiKzq7N3Lrdc
2022-10-26 01:36:23.849528 (p2p.Koinos) [node/node.go:374] : - /ip4/144.76.219.133/tcp/8888/p2p/QmYGwR34dRHqaU3ctpB2zmXZ3PJcUSGbE7mB3WyxkcsigH
2022-10-26 01:36:23.849955 (p2p.Koinos) [node/node.go:374] : - /ip4/161.35.57.92/tcp/8888/p2p/QmRG8ozdG6HVssPDvAMKaWniLmbpyLUo1B6DC6DHQXceD6
2022-10-26 01:36:23.850361 (p2p.Koinos) [node/node.go:374] : - /ip4/139.144.17.121/tcp/8888/p2p/QmcGiTpSm6YrmYo3rWoqrCPez2aJY4VdraBQsGsZKwFRuG
2022-10-26 01:36:28.131798 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmYGwR34dRHqaU3ctpB2zmXZ3PJcUSGbE7mB3WyxkcsigH, block application failed: block application timed out. Current error score: 5000
2022-10-26 01:36:28.133146 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmUwUmKLiQzMDg3xWkpfUnsbvBsrXK1uVqpiKzq7N3Lrdc, block application failed: block application timed out. Current error score: 5000
2022-10-26 01:36:28.133720 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmdrdcEnYDLgJUZzrhYNhYvEifBhiiTn7bndmnJUTma7dB, block application failed: block application timed out. Current error score: 5000
2022-10-26 01:36:28.134340 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmTmTvVcSQd73qXt6xGTrHwaEwjKZgaQ5ZkxYJzTkFfimH, block application failed: block application timed out. Current error score: 5000
2022-10-26 01:36:28.134780 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmV7q74bXxbg4yGY1kPgj9LsJPjgXqwc9SPtdTiSS67tvk, block application failed: block application timed out. Current error score: 5000
2022-10-26 01:36:28.135387 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmcGiTpSm6YrmYo3rWoqrCPez2aJY4VdraBQsGsZKwFRuG, block application failed: block application timed out. Current error score: 5000
2022-10-26 01:36:28.135997 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmRG8ozdG6HVssPDvAMKaWniLmbpyLUo1B6DC6DHQXceD6, block application failed: block application timed out. Current error score: 5000
2022-10-26 01:36:35.178021 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmRG8ozdG6HVssPDvAMKaWniLmbpyLUo1B6DC6DHQXceD6, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4957
2022-10-26 01:36:35.184530 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmV7q74bXxbg4yGY1kPgj9LsJPjgXqwc9SPtdTiSS67tvk, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4959
2022-10-26 01:36:35.190239 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmcGiTpSm6YrmYo3rWoqrCPez2aJY4VdraBQsGsZKwFRuG, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4959
2022-10-26 01:36:35.340881 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmdrdcEnYDLgJUZzrhYNhYvEifBhiiTn7bndmnJUTma7dB, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4958
2022-10-26 01:36:35.343245 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmYGwR34dRHqaU3ctpB2zmXZ3PJcUSGbE7mB3WyxkcsigH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4958
2022-10-26 01:36:35.346073 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmTmTvVcSQd73qXt6xGTrHwaEwjKZgaQ5ZkxYJzTkFfimH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4958
2022-10-26 01:36:35.346990 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmUwUmKLiQzMDg3xWkpfUnsbvBsrXK1uVqpiKzq7N3Lrdc, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4958
2022-10-26 01:36:42.249049 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmV7q74bXxbg4yGY1kPgj9LsJPjgXqwc9SPtdTiSS67tvk, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4918
2022-10-26 01:36:42.258006 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmRG8ozdG6HVssPDvAMKaWniLmbpyLUo1B6DC6DHQXceD6, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4915
2022-10-26 01:36:42.272926 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmcGiTpSm6YrmYo3rWoqrCPez2aJY4VdraBQsGsZKwFRuG, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4918
2022-10-26 01:36:45.848583 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmUwUmKLiQzMDg3xWkpfUnsbvBsrXK1uVqpiKzq7N3Lrdc, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4898
2022-10-26 01:36:45.849886 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmTmTvVcSQd73qXt6xGTrHwaEwjKZgaQ5ZkxYJzTkFfimH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4898
2022-10-26 01:36:45.851097 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmdrdcEnYDLgJUZzrhYNhYvEifBhiiTn7bndmnJUTma7dB, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4898
2022-10-26 01:36:45.875909 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmYGwR34dRHqaU3ctpB2zmXZ3PJcUSGbE7mB3WyxkcsigH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4898
2022-10-26 01:36:49.279487 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmV7q74bXxbg4yGY1kPgj9LsJPjgXqwc9SPtdTiSS67tvk, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4878
2022-10-26 01:36:49.312048 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmRG8ozdG6HVssPDvAMKaWniLmbpyLUo1B6DC6DHQXceD6, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4873
2022-10-26 01:36:49.328652 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmcGiTpSm6YrmYo3rWoqrCPez2aJY4VdraBQsGsZKwFRuG, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4878
2022-10-26 01:36:52.923595 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmTmTvVcSQd73qXt6xGTrHwaEwjKZgaQ5ZkxYJzTkFfimH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4858
2022-10-26 01:36:52.941467 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmdrdcEnYDLgJUZzrhYNhYvEifBhiiTn7bndmnJUTma7dB, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4858
2022-10-26 01:36:52.944380 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmUwUmKLiQzMDg3xWkpfUnsbvBsrXK1uVqpiKzq7N3Lrdc, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4858
2022-10-26 01:36:52.989515 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmYGwR34dRHqaU3ctpB2zmXZ3PJcUSGbE7mB3WyxkcsigH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4857
2022-10-26 01:36:56.307952 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmV7q74bXxbg4yGY1kPgj9LsJPjgXqwc9SPtdTiSS67tvk, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4838
2022-10-26 01:36:56.377385 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmcGiTpSm6YrmYo3rWoqrCPez2aJY4VdraBQsGsZKwFRuG, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4838
2022-10-26 01:36:56.391014 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmRG8ozdG6HVssPDvAMKaWniLmbpyLUo1B6DC6DHQXceD6, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4832
2022-10-26 01:36:59.995149 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmTmTvVcSQd73qXt6xGTrHwaEwjKZgaQ5ZkxYJzTkFfimH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4818
2022-10-26 01:37:00.029050 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmdrdcEnYDLgJUZzrhYNhYvEifBhiiTn7bndmnJUTma7dB, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4818
2022-10-26 01:37:00.034001 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmUwUmKLiQzMDg3xWkpfUnsbvBsrXK1uVqpiKzq7N3Lrdc, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4818
2022-10-26 01:37:00.108659 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmYGwR34dRHqaU3ctpB2zmXZ3PJcUSGbE7mB3WyxkcsigH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4815
2022-10-26 01:37:03.317539 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmV7q74bXxbg4yGY1kPgj9LsJPjgXqwc9SPtdTiSS67tvk, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4798
2022-10-26 01:37:03.401280 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmcGiTpSm6YrmYo3rWoqrCPez2aJY4VdraBQsGsZKwFRuG, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4798
2022-10-26 01:37:03.412375 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmRG8ozdG6HVssPDvAMKaWniLmbpyLUo1B6DC6DHQXceD6, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4792
2022-10-26 01:37:07.050973 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmTmTvVcSQd73qXt6xGTrHwaEwjKZgaQ5ZkxYJzTkFfimH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4778
2022-10-26 01:37:07.093851 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmdrdcEnYDLgJUZzrhYNhYvEifBhiiTn7bndmnJUTma7dB, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4778
2022-10-26 01:37:07.105347 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmUwUmKLiQzMDg3xWkpfUnsbvBsrXK1uVqpiKzq7N3Lrdc, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4777
2022-10-26 01:37:07.218637 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmYGwR34dRHqaU3ctpB2zmXZ3PJcUSGbE7mB3WyxkcsigH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4774
2022-10-26 01:37:10.344503 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmV7q74bXxbg4yGY1kPgj9LsJPjgXqwc9SPtdTiSS67tvk, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4759
2022-10-26 01:37:10.427959 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmRG8ozdG6HVssPDvAMKaWniLmbpyLUo1B6DC6DHQXceD6, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4751
2022-10-26 01:37:10.442541 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmcGiTpSm6YrmYo3rWoqrCPez2aJY4VdraBQsGsZKwFRuG, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4759
2022-10-26 01:37:14.122829 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmTmTvVcSQd73qXt6xGTrHwaEwjKZgaQ5ZkxYJzTkFfimH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4739
2022-10-26 01:37:14.181705 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmdrdcEnYDLgJUZzrhYNhYvEifBhiiTn7bndmnJUTma7dB, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4739
2022-10-26 01:37:14.203843 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmUwUmKLiQzMDg3xWkpfUnsbvBsrXK1uVqpiKzq7N3Lrdc, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4736
2022-10-26 01:37:14.330689 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmYGwR34dRHqaU3ctpB2zmXZ3PJcUSGbE7mB3WyxkcsigH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4734
2022-10-26 01:37:17.373983 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmV7q74bXxbg4yGY1kPgj9LsJPjgXqwc9SPtdTiSS67tvk, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4720
2022-10-26 01:37:17.491417 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmcGiTpSm6YrmYo3rWoqrCPez2aJY4VdraBQsGsZKwFRuG, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4720
2022-10-26 01:37:17.515117 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmRG8ozdG6HVssPDvAMKaWniLmbpyLUo1B6DC6DHQXceD6, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4708
2022-10-26 01:37:21.197533 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmTmTvVcSQd73qXt6xGTrHwaEwjKZgaQ5ZkxYJzTkFfimH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4700
2022-10-26 01:37:21.265948 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmdrdcEnYDLgJUZzrhYNhYvEifBhiiTn7bndmnJUTma7dB, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4700
2022-10-26 01:37:21.305787 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmUwUmKLiQzMDg3xWkpfUnsbvBsrXK1uVqpiKzq7N3Lrdc, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4697
2022-10-26 01:37:21.441031 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmYGwR34dRHqaU3ctpB2zmXZ3PJcUSGbE7mB3WyxkcsigH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4693
2022-10-26 01:37:22.517508 (p2p.Koinos) [p2p/connection_manager.go:179] : Connected to peer: /ip4/194.13.80.81/tcp/8888/p2p/QmPM6DZHzBrtRFnP3WK6rT8ShmMjv9t6Us3Z9tTQnKVAnQ
2022-10-26 01:37:23.046149 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmPM6DZHzBrtRFnP3WK6rT8ShmMjv9t6Us3Z9tTQnKVAnQ, peer does not have checkpoint block. Current error score: 243525
2022-10-26 01:37:23.047655 (p2p.Koinos) [p2p/connection_manager.go:213] : Disconnected from peer: /ip4/194.13.80.81/tcp/8888/p2p/QmPM6DZHzBrtRFnP3WK6rT8ShmMjv9t6Us3Z9tTQnKVAnQ
2022-10-26 01:37:23.811872 (p2p.Koinos) [node/node.go:370] : My address:
2022-10-26 01:37:23.813496 (p2p.Koinos) [node/node.go:371] : - /ip4/192.168.0.8/tcp/8888/p2p/QmYhPfRykBVTL2xnHH7igXuqVis4LRsxgbh4AmV2Hg1hpU
2022-10-26 01:37:23.814439 (p2p.Koinos) [node/node.go:372] : Connected peers:
2022-10-26 01:37:23.815596 (p2p.Koinos) [node/node.go:374] : - /ip4/144.76.219.133/tcp/8888/p2p/QmYGwR34dRHqaU3ctpB2zmXZ3PJcUSGbE7mB3WyxkcsigH
2022-10-26 01:37:23.816370 (p2p.Koinos) [node/node.go:374] : - /ip4/161.35.57.92/tcp/8888/p2p/QmRG8ozdG6HVssPDvAMKaWniLmbpyLUo1B6DC6DHQXceD6
2022-10-26 01:37:23.817095 (p2p.Koinos) [node/node.go:374] : - /ip4/139.144.17.121/tcp/8888/p2p/QmcGiTpSm6YrmYo3rWoqrCPez2aJY4VdraBQsGsZKwFRuG
2022-10-26 01:37:23.818327 (p2p.Koinos) [node/node.go:374] : - /ip4/139.144.17.130/tcp/8888/p2p/QmV7q74bXxbg4yGY1kPgj9LsJPjgXqwc9SPtdTiSS67tvk
2022-10-26 01:37:23.818949 (p2p.Koinos) [node/node.go:374] : - /ip4/64.62.202.13/tcp/8888/p2p/QmTmTvVcSQd73qXt6xGTrHwaEwjKZgaQ5ZkxYJzTkFfimH
2022-10-26 01:37:23.819560 (p2p.Koinos) [node/node.go:374] : - /ip4/185.130.47.36/tcp/8888/p2p/QmdrdcEnYDLgJUZzrhYNhYvEifBhiiTn7bndmnJUTma7dB
2022-10-26 01:37:23.820190 (p2p.Koinos) [node/node.go:374] : - /ip4/51.15.5.91/tcp/8888/p2p/QmUwUmKLiQzMDg3xWkpfUnsbvBsrXK1uVqpiKzq7N3Lrdc
2022-10-26 01:37:24.405949 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmV7q74bXxbg4yGY1kPgj9LsJPjgXqwc9SPtdTiSS67tvk, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4681
2022-10-26 01:37:24.538514 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmcGiTpSm6YrmYo3rWoqrCPez2aJY4VdraBQsGsZKwFRuG, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4681
2022-10-26 01:37:24.837941 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmRG8ozdG6HVssPDvAMKaWniLmbpyLUo1B6DC6DHQXceD6, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4668
2022-10-26 01:37:28.266498 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmTmTvVcSQd73qXt6xGTrHwaEwjKZgaQ5ZkxYJzTkFfimH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4661
2022-10-26 01:37:28.351574 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmdrdcEnYDLgJUZzrhYNhYvEifBhiiTn7bndmnJUTma7dB, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4661
2022-10-26 01:37:28.397597 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmUwUmKLiQzMDg3xWkpfUnsbvBsrXK1uVqpiKzq7N3Lrdc, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4658
2022-10-26 01:37:28.564029 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmYGwR34dRHqaU3ctpB2zmXZ3PJcUSGbE7mB3WyxkcsigH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4652
2022-10-26 01:37:31.461241 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmV7q74bXxbg4yGY1kPgj9LsJPjgXqwc9SPtdTiSS67tvk, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4643
2022-10-26 01:37:31.581404 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmcGiTpSm6YrmYo3rWoqrCPez2aJY4VdraBQsGsZKwFRuG, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4643
2022-10-26 01:37:31.858322 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmRG8ozdG6HVssPDvAMKaWniLmbpyLUo1B6DC6DHQXceD6, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4628
2022-10-26 01:37:35.322293 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmTmTvVcSQd73qXt6xGTrHwaEwjKZgaQ5ZkxYJzTkFfimH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4623
2022-10-26 01:37:35.412863 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmdrdcEnYDLgJUZzrhYNhYvEifBhiiTn7bndmnJUTma7dB, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4623
2022-10-26 01:37:35.466531 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmUwUmKLiQzMDg3xWkpfUnsbvBsrXK1uVqpiKzq7N3Lrdc, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4619
2022-10-26 01:37:35.761703 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmYGwR34dRHqaU3ctpB2zmXZ3PJcUSGbE7mB3WyxkcsigH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4613
2022-10-26 01:37:38.465365 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmV7q74bXxbg4yGY1kPgj9LsJPjgXqwc9SPtdTiSS67tvk, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4605
2022-10-26 01:37:38.629913 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmcGiTpSm6YrmYo3rWoqrCPez2aJY4VdraBQsGsZKwFRuG, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4605
2022-10-26 01:37:38.874175 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmRG8ozdG6HVssPDvAMKaWniLmbpyLUo1B6DC6DHQXceD6, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4589
2022-10-26 01:37:42.394460 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmTmTvVcSQd73qXt6xGTrHwaEwjKZgaQ5ZkxYJzTkFfimH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4585
2022-10-26 01:37:42.498349 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmdrdcEnYDLgJUZzrhYNhYvEifBhiiTn7bndmnJUTma7dB, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4585
2022-10-26 01:37:42.558897 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmUwUmKLiQzMDg3xWkpfUnsbvBsrXK1uVqpiKzq7N3Lrdc, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4581
2022-10-26 01:37:42.893539 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmYGwR34dRHqaU3ctpB2zmXZ3PJcUSGbE7mB3WyxkcsigH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4572
2022-10-26 01:37:45.496802 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmV7q74bXxbg4yGY1kPgj9LsJPjgXqwc9SPtdTiSS67tvk, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4567
2022-10-26 01:37:45.663721 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmcGiTpSm6YrmYo3rWoqrCPez2aJY4VdraBQsGsZKwFRuG, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4567
2022-10-26 01:37:46.076001 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmRG8ozdG6HVssPDvAMKaWniLmbpyLUo1B6DC6DHQXceD6, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4549
2022-10-26 01:37:49.468646 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmTmTvVcSQd73qXt6xGTrHwaEwjKZgaQ5ZkxYJzTkFfimH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4547
2022-10-26 01:37:49.583228 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmdrdcEnYDLgJUZzrhYNhYvEifBhiiTn7bndmnJUTma7dB, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4547
2022-10-26 01:37:49.652425 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmUwUmKLiQzMDg3xWkpfUnsbvBsrXK1uVqpiKzq7N3Lrdc, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4543
2022-10-26 01:37:50.033607 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmYGwR34dRHqaU3ctpB2zmXZ3PJcUSGbE7mB3WyxkcsigH, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4534
2022-10-26 01:37:52.529300 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmV7q74bXxbg4yGY1kPgj9LsJPjgXqwc9SPtdTiSS67tvk, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4530
2022-10-26 01:37:52.715807 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmcGiTpSm6YrmYo3rWoqrCPez2aJY4VdraBQsGsZKwFRuG, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4529
2022-10-26 01:37:56.521054 (p2p.Koinos) [p2p/error_handler.go:133] : Encountered peer error: QmRG8ozdG6HVssPDvAMKaWniLmbpyLUo1B6DC6DHQXceD6, local RPC error GetBlocksByID, rpc failed, context deadline exceeded. Current error score: 4489

Answer 4 · 2022-10-26T15:18:00.000Z

Made some progress. The process stopped again just now with exit code 137 possibly indicating the container was out of memory.

docker inspect shows the out of memory flag (OOMKilled) as false. All the same I strongly suspect the container stopped for this reason. 8GB is currently allocated to docker.

"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2022-10-26T14:35:49.550718477Z",
"FinishedAt": "2022-10-26T15:00:20.521933859Z"
},

Answer 5 · 2022-10-27T23:18:13.000Z

Are you running all of the microservices for an API node?

Answer 6 · 2022-10-27T23:26:09.000Z

Yes, and docker stats does show block_store ballooning up to around 3GB when syncing some bigger blocks. I wondered if there may be a memory leak, but it was almost entirely released after the node was synced. Have not seen this service exit unexpectedly after increasing Docker’s ram allocation to 14GB.