vmagamedov/grpclib

Request was cancelled: Connection lost / RST_STREAM

NuSkooler opened this issue · 1 comments

I'm running into a issue with STREAM_UNARY request in my server: When sending many messages the server side unexpectedly disconnects giving the client side a RST_STREAM.

This appears to be related to the sum of the message sizes. For example, I can send an identical message (one per client side write) a few hundred times before the issue occurs. If I reduce the size of the message I can send thousands before the issue occurs. No exception is thrown. This all happens within 1-2 seconds at most (no timeouts are set on contexts).

Server Example:

async def MyCall(
        self,
        stream: [
            MessageRequest,
            MessageResponse,
        ],
    ):
        is_init = False
        count = 0

        async for payload in stream:
            if not is_init:
                # ...process "Hello"
                is_init = True
            else:
                # ...do stuff with a payload
                count += 1

        # I don't actually get here
        await stream.send_message(MessageResponse())

And the client C++ side:

auto writer = getWriter();

//  Send the Hello
MessageRequest message;
auto& hello = *message.mutable_hello();

if (writer->Write(message))
{
    //  ...now stream payloads
    message.clear_hello();
    for (const auto& payload : payloads)
    {
        *message.mutable_payload() = payload;
        if (!writer->Write(message))
        {
            break;
        }
    }
}

writer->WritesDone();
const auto status = writer->Finish();    

Message sizes I'm using in above example:

  • ~4K I get 200-220 in before the issue occurs
  • ~850 bytes I get about 1200 in

Versions:
grpclib version 0.4.1
gRPC C++ 1.31.1
Python 3.6

Any ideas?

Disregard. This is not a grpclib issue at all: Our connections go through nginx and the particular location was not setting client_max_body_size -- this causes nginx to cut the stream at 1 MiB by default.