Error: goaway Frame with Invalid Size when trying gRPC Request

[sigridjineth]

Symptoms

• When trying to send requests to gRPC endpoints for the evmos blockchain (using https://grpc.bd.evmos.org:9090), I encountered with the following error.
  connection error: hyper::Error(Http2, Error { kind: GoAway(b"", FRAME_SIZE_ERROR, Library) })
• I wonder what is the root cause of the error and how to deal with the error. thanks.
• The endpoint works well on the terminal

export NODE=grpc.bd.evmos.org:9090
grpcurl -d '{"'address'": "'$EVMOS_ADDR_3'"}' $NODE cosmos.bank.v1beta1.Query/AllBalances

{
  "balances": [
    {
      "denom": "aevmos",
      "amount": "19254711178107386863"
    },
    {
      "denom": "ibc/ED07A3391A112B175915CD8FAF43A2DA8E4790EDE12566649D0C2F97716B8518",
      "amount": "63743"
    }
  ],
  "pagination": {
    "total": "2"
  }
}

Error Stack

/Users/sigridjin.eth/Documents/github/ParachuteDrop-LightClient/target/debug/parachutedrop-rust-server
[2022-12-26T03:01:44Z INFO  actix_server::builder] Starting 10 workers
[2022-12-26T03:01:44Z INFO  actix_server::server] Actix runtime found; starting in Actix runtime
[2022-12-26T03:01:51Z DEBUG hyper::client::connect::dns] resolving host="grpc.bd.evmos.org"
[2022-12-26T03:01:51Z DEBUG hyper::client::connect::http] connecting to 18.211.85.17:9090
[2022-12-26T03:01:51Z DEBUG hyper::client::connect::http] connected to 18.211.85.17:9090
[2022-12-26T03:01:51Z DEBUG h2::client] binding client connection
[2022-12-26T03:01:51Z DEBUG h2::client] client connection bound
[2022-12-26T03:01:51Z DEBUG h2::codec::framed_write] send frame=Settings { flags: (0x0), enable_push: 0, initial_window_size: 2097152, max_frame_size: 16384 }
[2022-12-26T03:01:51Z DEBUG h2::proto::connection] Connection; peer=Client
[2022-12-26T03:01:51Z DEBUG h2::codec::framed_write] send frame=WindowUpdate { stream_id: StreamId(0), size_increment: 5177345 }
[2022-12-26T03:01:59Z DEBUG tower::buffer::worker] service.ready=true message=processing request
[2022-12-26T03:01:59Z DEBUG h2::codec::framed_write] send frame=Headers { stream_id: StreamId(1), flags: (0x4: END_HEADERS) }
[2022-12-26T03:01:59Z DEBUG h2::codec::framed_write] send frame=Data { stream_id: StreamId(1) }
[2022-12-26T03:01:59Z DEBUG h2::codec::framed_write] send frame=Data { stream_id: StreamId(1), flags: (0x1: END_STREAM) }
[2022-12-26T03:02:03Z DEBUG h2::proto::connection] Connection::poll; connection error error=GoAway(b"", FRAME_SIZE_ERROR, Library)
[2022-12-26T03:02:03Z DEBUG h2::codec::framed_write] send frame=GoAway { error_code: FRAME_SIZE_ERROR, last_stream_id: StreamId(0) }
[2022-12-26T03:02:03Z DEBUG h2::proto::connection] Connection::poll; connection error error=GoAway(b"", FRAME_SIZE_ERROR, Library)
[2022-12-26T03:02:03Z DEBUG hyper::proto::h2::client] connection error: broken pipe
[2022-12-26T03:02:03Z DEBUG hyper::client::service] connection error: hyper::Error(Http2, Error { kind: GoAway(b"", FRAME_SIZE_ERROR, Library) })
[2022-12-26T03:02:03Z DEBUG hyper::proto::h2::client] client response error: connection error detected: frame with invalid size

Code

#[get("/evmos")]
pub async fn evmos_health() -> Result<HttpResponse, HTTPError> {
    // grpcurl -d '{"'address'": "'$EVMOS_ADDR_3'"}' $NODE cosmos.bank.v1beta1.Query/AllBalances
    let endpoint = tonic::transport::Endpoint::new("https://grpc.bd.evmos.org:9090"
            .parse::<tonic::transport::Uri>().unwrap())
            .unwrap();
    let channel = endpoint.connect().await;
    let mut connected_client = QueryClient::new(channel.unwrap());
    let request = QueryAllBalancesRequest {
        address: "evmos1f57x5pm9wpvlu4qldy864j0hdnq025287r65z2".to_string(),
        pagination: Option::from(PageRequest {
            key: Vec::from("".to_string()),
            offset: 0,
            limit: 0,
            count_total: false,
            reverse: false
        }),
    };
    let response = connected_client.all_balances(request).await;

    // let response = connected_client.all_balances(request).await;
    // response.unwrap();
    Ok(HttpResponse::Ok().json(HealthResponse {
        status: StatusCode::OK.as_u16(),
        message: "OK".to_string(),
        data: None,
    }))
}

[tony-iqlusion]

Hi there!

This is an error I'm assuming is arising in the generated code of cosmos-sdk-proto? (perhaps cosmos_sdk_proto::cosmos::bank::v1beta1::QueryAllBalancesRequest?)

Whatever the cause is, it's something out of our control, arising in the low-level HTTP/2 client.

Either there's some error with https://grpc.bd.evmos.org:9090 or a bug in Tonic, the gRPC library we use.

I would suggest trying to file this issue against Tonic and see if they might be able to assist: https://github.com/hyperium/tonic

[Philipp-Sc]

@sigridjineth Did you find a solution for this issue? I am running into the same error with a very similar setup.

[tony-iqlusion]

Perhaps file an upstream error against Tonic? https://github.com/hyperium/tonic

[tony-iqlusion]

I don't think there's anything we can do here. Whatever the solution to this issue is needs to happen upstream from this project. Closing.