ResourceExhausted: grpc: received message larger than max

Question

ResourceExhausted: grpc: received message larger than max

deni64k opened this issue 4 years ago · 8 comments

Hello,
I'm syncing a Prysm with a database using chaind, and I get these errors persistently.
I tried to hack MaxCallRecvMsgSize, but no luck.

Dec 16 16:22:26 chaind[110813]: {"level":"warn","service":"proposerduties","impl":"standard","epoch":8,"error":"failed to fetch proposer duties: call to GetDuties() failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5557752 vs. 4194304)","time":"2020-12-16T16:22:26Z","message":"Failed to update proposer duties"}
Dec 16 16:27:56 chaind[110813]: {"level":"warn","service":"proposerduties","impl":"standard","epoch":9,"error":"failed to fetch proposer duties: call to GetDuties() failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5557752 vs. 4194304)","time":"2020-12-16T16:27:56Z","message":"Failed to update proposer duties"}
Dec 16 16:32:40 chaind[110813]: {"level":"warn","service":"proposerduties","impl":"standard","epoch":10,"error":"failed to fetch proposer duties: call to GetDuties() failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5557752 vs. 4194304)","time":"2020-12-16T16:32:40Z","message":"Failed to update proposer duties"}
Dec 16 16:38:04 chaind[110813]: {"level":"warn","service":"proposerduties","impl":"standard","epoch":11,"error":"failed to fetch proposer duties: call to GetDuties() failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5557752 vs. 4194304)","time":"2020-12-16T16:38:04Z","message":"Failed to update proposer duties"}

Answer 1 · 2020-12-16T21:00:35.000Z

Can you try running prysm with the following additional options:

--rpc-max-page-size=100000 --grpc-max-msg-size=268435456

These should increase the maximum GRPC message size such that it can return the required data.

Answer 2 · 2020-12-16T22:07:55.000Z

Just tried. Still getting the same error.

Answer 3 · 2020-12-16T22:30:59.000Z

chaind version — tried the latest master and v0.1.4.
Prysm is v1.0.5.
The network is Pyrmont.

Answer 4 · 2020-12-17T08:24:34.000Z

Though, just --grpc-max-msg-size=268435456 seemed to help. It's not clear yet.

Answer 5 · 2020-12-17T17:40:48.000Z

The --grpc-max-msg-size sets the maximum size of a single message. The --rpc-max-page-size sets the maximum number of entries returned per page for paged APIs. Both are required to obtain information from prysm. What error are you seeing with both of these flags enabled?

Answer 6 · 2020-12-17T20:01:40.000Z

It's the same error when both flags are on.

Answer 7 · 2020-12-17T20:53:55.000Z

Please could you post your logs with these flags enabled? Specifically, the message:

Dec 16 16:22:26 chaind[110813]: {"level":"warn","service":"proposerduties","impl":"standard","epoch":8,"error":"failed to fetch proposer duties: call to GetDuties() failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5557752 vs. 4194304)","time":"2020-12-16T16:22:26Z","message":"Failed to update proposer duties"}

states the maximum message size 4194304, which is the default. If grpc-max-msg-size is set then this value should change.

Answer 8 · 2020-12-18T02:40:56.000Z

Sorry, we've already switched to beaconchain-explorer for importing the chain into a database. Thanks for the hints!