Twirp

Twirp allows to easily define RPC services and clients that communicate using Protobuf or JSON over HTTP.

The Twirp protocol is implemented in multiple languages. This means that you can write your service in one language and automatically generate clients in other languages. Refer to the Golang implementation for more details on the project.

Install

Add gem "twirp" to your Gemfile, or install with

gem install twirp

Code generation works with protoc (the protobuf compiler) using the --ruby_out option to generate messages and --twirp_ruby_out to generate services and clients. Make sure to install protoc version 3+.

Then use go get (Golang) to install the ruby_twirp protoc plugin:

go get -u github.com/cyrusaf/ruby-twirp/protoc-gen-twirp_ruby

Code Generation

Service and client definitions can be auto-generated form a .proto file. For example, given a Protobuf file like example/hello_world/service.proto, you can auto-generate proto and twirp files with the command:

protoc --proto_path=. --ruby_out=. --twirp_ruby_out=. ./example/hello_world/service.proto

Service DSL

The generated code makes use of the service DSL. For example, the generated code for the HelloWorld service looks like this:

module Example
  class HelloWorldService < Twirp::Service
    package "example"
    service "HelloWorld"
    rpc :Hello, HelloRequest, HelloResponse, :ruby_method => :hello
  end

  class HelloWorldClient < Twirp::Client
    client_for HelloWorldService
  end
end

The HelloRequest and HelloResponse messages are expected to be google-protobuf messages. They are generated by protoc, but could also be defined from their DSL.

Twirp Service Handler

A handler implements the rpc methods. For example a handler for HelloWorldService:

class HelloWorldHandler

  def hello(req, env)
    if req.name.empty?
      return Twirp::Error.invalid_argument("is mandatory", argument: "name")
    end

    {message: "Hello #{req.name}"}
  end

end

For each rpc method:

The req argument is the input request message, already serialized.
The env argument is the Twirp environment with data related to the request (e.g. env[:output_class]), and other fields that could have been set from before-hooks (e.g. env[:user_id] from authentication).
The returned value is expected to be the response message (or its attributes), or a Twirp::Error.

Start the Service

Instantiate the service with your handler impementation. The service is a Rack app. For example:

handler = HelloWorldHandler.new()
service = Example::HelloWorldService.new(handler)

require 'rack'
Rack::Handler::WEBrick.run service

Rack apps can also be mounted as Rails routes (e.g. mount service, at: service.full_name) and are compatible with many other HTTP frameworks.

Unit Tests

Twirp already takes care of HTTP routing and serialization, you don't really need to test that part, insteadof that, focus on testing the handler using the method .call_rpc(rpc_method, attrs={}, env={}) on the service:

require 'minitest/autorun'

class HelloWorldHandlerTest < Minitest::Test

  def test_hello_responds_with_name
    resp = service.call_rpc :Hello, name: "World"
    assert_equal "Hello World", resp.message
  end

  def test_hello_name_is_mandatory
    twerr = service.call_rpc :Hello, name: ""
    assert_equal :invalid_argument, twerr.code
  end

  def service
    handler = HelloWorldHandler.new()
    Example::HelloWorldService.new(handler)
  end
end

Twirp Clients

Instantiate a client with the service base url:

client = Example::HelloWorldClient.new("http://localhost:3000")

Clients implement the same methods as the service handler. For example the client for HelloWorldService implements the hello method:

resp = client.hello(name: "World")

As an alternative, in case that a service method collides with a Ruby method, you can always use the more general .rpc method:

resp = client.rpc(:Hello, name: "World") # alternative

If the request fails, the response has an error with a Twirp::Error. If the request succeeds, the response has data with an instance of the response message class.

if resp.error
  puts resp.error # <Twirp::Error code:... msg:"..." meta:{...}>
else
  puts resp.data  # <Example::HelloResponse: message:"Hello World">
end

Configure Clients with Faraday

While Twirp takes care of routing, serialization and error handling, other advanced HTTP options can be configured with Faraday middleware. Clients can be initialized with a Faraday connection. For example:

conn = Faraday.new(:url => 'http://localhost:3000') do |c|
  c.use Faraday::Request::Retry
  c.use Faraday::Request::BasicAuthentication, 'login', 'pass'
  c.use Faraday::Response::Logger # log to STDOUT
  c.use Faraday::Adapter::NetHttp # can use different HTTP libraries
end

client = Example::HelloWorldClient.new(conn)

Protobuf or JSON

Protobuf is used by default. To serialize with JSON, set the content_type option as 2nd argument:

client = Example::HelloWorldClient.new(conn, content_type: "application/json")
resp = client.hello(name: "World") # serialized with JSON

Add-hoc JSON requests

If you just want to make a few quick requests from the console, you can make a ClientJSON instance. This doesn't require a service definition at all, but in the other hand, request and response values are not validated. Responses are just a Hash with attributes.

client = Twirp::ClientJSON.new(conn, package: "example", service: "HelloWorld")
resp = client.rpc(:Hello, name: "World") # serialized with JSON, resp.data is a plain Hash

Server Hooks

In the lifecycle of a server request, Twirp starts by routing the request to a valid RPC method. If routing fails, the on_error hook is called with a bad_route error. If routing succeeds, the before hook is called before calling the RPC method handler, and then either on_success or on_error depending if the response is a Twirp error or not.

routing -> before -> handler -> on_success
                             -> on_error

On every request, one and only one of on_success or on_error is called.

If exceptions are raised, the exception_raised hook is called. The exceptioni is wrapped with an internal Twirp error, and if the on_error hook was not called yet, then it is called with the wrapped exception.

routing -> before -> handler
                     ! exception_raised -> on_error

Hooks are setup in the service instance. For example:

svc = Example::HelloWorldService.new(handler)

svc.before do |rack_env, env|
  # Runs if properly routed to an rpc method, but before calling the method handler.
  # This is the only place to read the Rack env to access http request and middleware data.
  # The Twirp env has the same routing info as in the handler method, e.g. :rpc_method, :input and :input_class.
  # Returning a Twirp::Error here cancels the request, and the error is returned instead.
  # If an exception is raised, the exception_raised hook will be called followed by on_error.
  env[:user_id] = authenticate(rack_env)
end

svc.on_success do |env|
  # Runs after the rpc method is handled, if it didn't return Twirp errors or raised exceptions.
  # The env[:output] contains the serialized message of class env[:ouput_class].
  # If an exception is raised, the exception_raised hook will be called.
  success_count += 1
end

svc.on_error do |twerr, env|
  # Runs on error responses, that is:
  #  * bad_route errors
  #  * before filters returning Twirp errors or raising exceptions.
  #  * hander methods returning Twirp errors or raising exceptions.
  # Raised exceptions are wrapped with Twirp::Error.internal_with(e).
  # If an exception is raised here, the exception_raised hook will be called.
  error_count += 1
end

svc.exception_raised do |e, env|
  # Runs if an exception was raised from the handler or any of the hooks.
  puts "[Error] #{e}\n#{e.backtrace.join("\n")}"
end

KMontag42/ruby-twirp