Why grpc-gateway mode is base64?

[sulliwane]

Hello,

I'm wondering why the nginx-gateway is responding with base64 strings, instead of directly sending binary protobuf messages over the wire?

And is there a way to change the mode=base64 to mode=binary when compiling the gateway (it that makes sense of course)

Thank you :)

[stanley-cheung]

Two major reasons:

1. To support streaming, we need to be able to read the partial data being sent from the server (with the Transfer-Encoding: chunked HTTP header). But for now, the underlying XHR implementation does not support reading partial binary data. See https://hpbn.co/xmlhttprequest/

Partial response can be read only from the responseText attribute, which limits us to text-only transfers. There is no way to read partial response of a binary transfer.

2. We sent trailing metadata as a separate message at the end, basically using the same Transfer-Encoding: chunked mechanism. We don't want to have to wait for the entire trailers to be sent before the client can read the 'response'.

[sulliwane]

Ok, the reading about XHR is interesting, I understand better now.

Regarding your second point, could you be more specific when saying

"sending trailing metadata as a separate message at the end"

Because in this other thread, I'm trying to manually decode the base64 response to a protobuf message. I correctly receive the whole response (as a base64 string), but when deserizalizing, I get all the data in the same Message field, with many scrambled character...

Could it be the result of these metadata being mixed with the protobuf message response?

Many thanks!

[mwitkow]

@stanley-cheung please talk to @MarcusImprobable. We solved it using xhr.responseType = "text" and this magical byte mapping hack:
https://github.com/improbable-eng/grpc-web/blob/master/ts/src/transports/xhr.ts#L4

Seems to work on Safari, which is the new IE ;)

[wenbozhu]

@sulliwane The current github release is a pre-alpha release, with the core shared with our internal release. If you don't mind, let's focus on the API (feedback) rather than the implementation detail for the JS client (which will be updated frequently before beta).

Thanks for all the feedback!

===

@mwitkow Yes, I have looked at this and we will evaluate the xhr hack which would be a great optimization. We also need make grpc-web client works across all browsers (inc. mobile browsers).

Re: base64, it is also required for serving binary responses from cookie domains (to prevent XSS etc).