gRPC-client-nginx-module
This module is experimental.
Install
First of all, build this module into your OpenResty:
./configure ... \
--with-threads \
--with-cc-opt="-DNGX_GRPC_CLI_ENGINE_PATH=/path/to/libgrpc_engine.so" \
--add-module=/path/to/grpc-client-nginx-module
We need to specify the path of engine via build argument "-DNGX_GRPC_CLI_ENGINE_PATH".
Then, compile the gRPC engine:
cd ./grpc-engine && go build -o libgrpc_engine.so -buildmode=c-shared main.go
After that, install the Lua rock:
luarocks install grpc-client-nginx-module
Make sure the Lua rock version matches the tag version of the Nginx module.
Finally, setup the thread pool:
# Only one background thread is used to communicate with the gRPC engine
thread_pool grpc-client-nginx-module threads=1;
http {
...
}
stream {
...
}
Synopsis
access_by_lua_block {
-- we can't require this library in the init_by_lua
local gcli = require("resty.grpc")
assert(gcli.load("t/testdata/rpc.proto")) -- load proto definition into the library
-- connect the target
local conn = assert(gcli.connect("127.0.0.1:2379", {insecure = true}))
-- send unary request
local res = assert(conn:call("etcdserverpb.KV", "Put", {key = 'k', value = 'v'}))
}
This module can be run in stream subsystem too:
preread_by_lua_block {
local gcli = require("resty.grpc")
assert(gcli.load("t/testdata/rpc.proto"))
local conn = assert(gcli.connect("127.0.0.1:2379"))
local res = conn:call("etcdserverpb.KV", "Put", {key = 'k', value = 'v'})
conn:close()
}
return '';
Method
load
syntax: ok, err = resty.grpc.load(proto[, proto_type])
Load the definition of the given proto, which can be used later.
The proto_type
can be one of:
PROTO_TYPE_FILE
PROTO_TYPE_STR
For instance, grpc.load("t/testdata/rpc.proto", grpc.PROTO_TYPE_FILE)
If not given, PROTO_TYPE_FILE
will be used by default.
connect
syntax: conn, err = resty.grpc.connect(target, connectOpt)
Create a gRPC connection
connectOpt:
insecure
: whether connecting the target in an insecure(plaintext) way. True by default. Set it to false will use TLS connection.tls_verify
: whether to verify the server's TLS certificate.max_recv_msg_size
: sets the maximum message size in bytes the client can receive.client_cert
: the path of certificate used in client certificate verification.client_key
: the path of key used in client certificate verification. The key should match the given certificate.trusted_ca
: the path of trusted certificate.
call
syntax: res, err = conn:call(service, method, request, callOpt)
Send a unary request.
The request
is a Lua table that will be encoded according to the proto.
The res
is a Lua table that is decoded from the proto message.
callOpt:
timeout
: Set the timeout value in milliseconds for the whole call. 60000 milliseconds by default.int64_encoding
: Set the eocoding for int64. The value can be one of:- INT64_AS_NUMBER: set value to integer when it fit into uint32, otherwise return a number (default)
- INT64_AS_STRING: same as above, but return a string instead
- INT64_AS_HEXSTRING: same as above, but return a hexadigit string instead
For example,
conn:call("etcdserverpb.KV", "Put", {key = 'k', value = 'v'},
{int64_encoding = gcli.INT64_AS_STRING})
will decode int64 result in #number
.
Note: The string returned by int64_as_string
or int64_as_hexstring
will prefix a '#'
character.
This behavior is done in lua-protobuf
.
metadata
: Set the gRPC metadata with an array of key-value pairs.
For example,
conn:call("etcdserverpb.KV", "Put", {key = 'k', value = 'v'},
{metadata = {
{"user", "john"},
{"password", "*&()"},
{"key", "val1"},
{"key", "val2"},
}})
new_server_stream
syntax: stream, err = conn:new_server_stream(service, method, request, callOpt)
Create a server stream.
The request
is a Lua table that will be encoded according to the proto.
The stream
is the server stream which can be used to read the data.
callOpt:
timeout
: Set the timeout value in milliseconds for the whole lifetime of the stream. 60000 milliseconds by default.int64_encoding
: Set the eocoding for int64.metadata
: Set the gRPC metadata with an array of key-value pairs.
recv
syntax: res, err = stream:recv()
Receive a response from the stream.
The res
is a Lua table that is decoded from the proto message.
new_client_stream
syntax: stream, err = conn:new_client_stream(service, method, request, callOpt)
Create a client stream.
The request
is a Lua table that will be encoded according to the proto.
The stream
is the client stream which can be used to send/recv the data.
callOpt:
timeout
: Set the timeout value in milliseconds for the whole lifetime of the stream. 60000 milliseconds by default.int64_encoding
: Set the eocoding for int64.
send
syntax: ok, err = stream:send(request)
Send a request via the stream.
The request
is a Lua table that will be encoded according to the proto.
recv_close
syntax: res, err = stream:recv_close()
Receive a response from the stream and close the stream.
The res
is a Lua table that is decoded from the proto message.
new_bidirectional_stream
syntax: stream, err = conn:new_bidirectional_stream(service, method, request, callOpt)
Create a bidirectional stream.
The request
is a Lua table that will be encoded according to the proto.
The stream
is the client stream which can be used to send/recv the data.
callOpt:
timeout
: Set the timeout value in milliseconds for the whole lifetime of the stream. 60000 milliseconds by default.int64_encoding
: Set the eocoding for int64.
The bidirectional stream has send
and recv
, which are equal to the corresponding
version in client/server streams.
close_send
syntax: ok, err = stream:close_send()
Close the send side of the bidirectional stream.
Why don't we
Why don't we use the gRPC code in Nginx
Because Nginx doesn't provide an interface for the gRPC feature. It requires we to copy & paste and assemble the components.
Why don't we compile the gRPC library into the Nginx module
The official gRPC library is written in C++ and requires >2GB dependencies. We don't use bazel/cmake to build Nginx and downloading >2GB dependencies will slow down our build process.
Therefore we choose gRPC-go as the core of gRPC engine. If the performance is an issue, we may consider using Rust instead.
Debug
Because go's scheduler doesn't work after fork
, we have to load the Go library
at runtime in each worker.
As a consequence of that, we can't use dlv to debug the process. Therefore we need
to use log debug
. All logs printed by the std log library will be written to file
/tmp/grpc-engine-debug.log
.
Limitation
- We require this Nginx module runs on 64bit aligned machine, like x64 and arm64.
- resolve
import
in the loaded.proto
file (we can handle it like what we have done in Apache APISIX) - very big integer in int64 can't be displayed correctly due to the missing int64 support in LuaJIT.