google/ml-metadata

Support for additional grpc options other than max_receive_message_length?

mwakaba2 opened this issue · 7 comments

mwakaba2 commented

Hi there!

I'd like to pass in a particular grpc option to the servers, but it seems like I can't pass in any other custom grpc options?

("grpc.http2.max_ping_strikes", 0)

https://github.com/google/ml-metadata/blob/r1.0.0/ml_metadata/metadata_store/metadata_store.py#L138

Is there a way to pass in other options other than the ones listed here?
https://github.com/google/ml-metadata/blob/r1.0.0/ml_metadata/proto/metadata_store.proto#L605

// A list of supported GRPC arguments defined in:
// https://grpc.github.io/grpc/core/group__grpc__arg__keys.html
message GrpcChannelArguments {
  // Maximum message length in bytes per response that the channel can receive.
  optional int64 max_receive_message_length = 1;
}
hughmiao commented

the proto options need to be extended. qq, what's the deployment setting and problem you are facing? does max_ping_strikes resolve your problem?

mwakaba2 commented

Hi @hughmiao, We have two production grpc servers with MLMD (V1.0.0). Our mlmd client is on V0.26.0 because we're blocked on TFX upgrade that depends on a Tensorflow upgrade that's in progress.

We've been getting several of "too many pings" errors when connecting to mlmd.

...install/grpcio-1.38.0-cp37-cp37m-linux_x86_64.whl.88d83fe88ca959b5bfa75fe1d1a12dad51dc3cfe/grpcio-1.38.0-cp37-cp37m-linux_x86_64.whl/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "Too many pings"
	debug_error_string = "{"created":"@1628030317.191658028","description":"Error received from peer ipv4:10.59.16.60:443","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"Too many pings","grpc_status":14}"
>

So we wanted to set it to ("grpc.http2.max_ping_strikes", 0) instead of the default 2 pings.
0 means no limit.

hughmiao commented

got it. thanks for the contexts. will add it to the options.

/cc @BrianSong

mwakaba2 commented

thank you @hughmiao, Can these options be available for version 1.0.0? I'm not able to upgrade to v1.1.0 yet.

hughmiao commented

The fix will be available in 1.2.0, which will be released later this week. Consider to pull the change and build wheel from source for older version [1]

[1] https://github.com/google/ml-metadata#3-build-the-pip-package

  • 👍1
mwakaba2 commented

@hughmiao thank you for working on this quickly!

hughmiao commented

np. closing the issue, the 1.2.0 is released.