A protobuf RPC framework for JavaScript and TypeScript
TwirpScript is an implementation of Twirp. TwirpScript autogenerates Javascript or TypeScript clients and servers from protocol buffers. The generated clients can be used in browser or Node.js runtimes, enabling type safe communication between web applications and a server or server-to-server. TwirpScript implements the latest Twirp Wire Protocol (v7).
Language | Clients | Servers |
---|---|---|
JavaScript | ✅ | ✅ |
Typescript | ✅ | ✅ |
- Overview
- Highlights 🛠
- Installation 📦
- Getting Started
- Configuration 🛠
- JSON
- Examples 🚀
- Working with other tools
- Caveats, Warnings and Issues
⚠️ - FAQ
- Contributing 👫
- Licensing 📃
Twirp is a simple RPC framework built on protocol buffers. TwirpScript generates JavaScript or TypeScript clients and servers from .proto
service specifications. The generated clients can be used in the browser or in Node.js runtimes. This enables type safe communication between the client and server, as well as reduced payload sizes when using protobuf
as the serialization format.
You define your service in a .proto
specification file, and TwirpScript will generate client and service handlers for that service. You fill in the business logic that powers the server, and TwirpScript handles the boilerplate.
To learn more about the motivation behind Twirp (and a comparison to REST APIs and gRPC), check out the announcement blog.
-
Isomorphic. TwirpScript's generated serializers/deserializers can be consumed in the browser or Node.js runtimes.
-
Small. TwirpScript's runtime and generated code are built with tree shaking to minimize bundle sizes. This results in a significantly smaller bundle size than google-protobuf. TwirpScript's runtime is 2KB (1.2 gzipped). The serialization runtime, ProtoScript, is 37KB (7.2 gzipped). ProtoScript will be eliminated from bundles when only using the generated JSON clients.
-
In-editor API documentation. Comments in your
.proto
files become TSDoc comments in the generated code and will show inline documentation in supported editors. -
Idiomatic JavaScript / TypeScript code. None of the Java idioms that
protoc --js_out
generates such as theList
suffix naming for repeated fields,Map
suffix for maps, or the various getter and setter methods. TwirpScript generates and consumes plain JavaScript objects over classes.
-
Install the protocol buffers compiler:
MacOS:
brew install protobuf
Linux:
apt install -y protobuf-compiler
Windows:
choco install protoc
Or install from a precompiled binary.
-
Add this package to your project:
npm install twirpscript
oryarn add twirpscript
- Define your service in a
.proto
file. - Run
npx twirpscript
to generate JavaScript or TypeScript code from your.proto
file. This will generate JSON and Protobuf clients, a service interface, and service utilities. - If you only need a client, you're done! Use the generated client to make requests to your server.
- Implement the generated service interface.
- Add your implemented service to your application server's routes.
Create a proto
specification file:
src/protos/haberdasher.proto
syntax = "proto3";
// Haberdasher service makes hats for clients.
service Haberdasher {
// MakeHat produces a hat of mysterious, randomly-selected color!
rpc MakeHat(Size) returns (Hat);
}
// Size of a Hat, in inches.
message Size {
int32 inches = 1; // must be > 0
}
// A Hat is a piece of headwear made by a Haberdasher.
message Hat {
int32 inches = 1;
string color = 2; // anything but "invisible"
string name = 3; // i.e. "bowler"
}
This will generate haberdasher.pb.ts
(or haberdasher.pb.js
for JavaScript users) in the same directory as as haberdasher.proto
. Any comments will become TSDoc comments and will show inline in supported editors.
npx twirpscript
will compile all.proto
files in your project.
Use the generated clients to make json
or protobuf
requests to your server:
src/client.ts
import { client } from "twirpscript";
import { MakeHat } from "../protos/haberdasher.pb";
client.baseURL = "http://localhost:8080";
const hat = await MakeHat({ inches: 12 });
console.log(hat);
The above client code may be used in browser or node.js runtimes. See a Node.js client example.
If you have an existing Twirp server you're connecting to and only need a client, that's it! You're done. If you're implementing a service as well, keep reading.
src/server/haberdasher/index.ts
import { Haberdasher, createHaberdasher } from "../../protos/haberdasher.pb";
const haberdasher: Haberdasher = {
MakeHat: (size) => {
return {
inches: size.inches,
color: "red",
name: "fedora",
};
},
};
export const haberdasherHandler = createHaberdasher(haberdasher);
src/server/index.ts
import { createServer } from "http";
import { createTwirpServer } from "twirpscript";
import { haberdasherHandler } from "./haberdasher";
const PORT = 8080;
const app = createTwirpServer([haberdasherHandler]);
createServer(app).listen(PORT, () =>
console.log(`Server listening on port ${PORT}`)
);
If you're deploying to a serverless environment such as AWS Lambda, replace createTwirpServer
above with createTwirpServerless
. See the aws lambda example for a full project!
Clients can be configured globally, at the RPC callsite, or with middleware. The order of precedence is middleware > call site configuration > global configuration. Middleware overrides call site configuration, and call site configuration overrides global configuration.
Name | Description | Type | Example |
---|---|---|---|
baseURL | The base URL for the RPC. The service path will be appended to this string. | string | "https://my.server.com/" |
headers | HTTP headers to include in the RPC. | Record<string, string> | { "idempotency-key": "foo" } |
prefix | A path prefix such as "/my/custom/prefix". Defaults to "/twirp", but can be set to "". | string | "/my/custom/prefix" |
rpcTransport | The transport to use for the RPC. Defaults to fetch . Overrides must conform to a subset of the fetch interface defined by the RpcTransport type. |
RpcTransport |
fetch |
src/client.ts
import { MakeHat } from "./protos/haberdasher.pb";
const hat = await MakeHat({ inches: 12 }, { baseURL: "http://localhost:8080" });
console.log(hat);
baseURL
can be globally configured, instead of providing it for every RPC call site:
import { client } from "twirpscript";
// http://localhost:8080 is now the default `baseURL` for _all_ TwirpScript RPCs
client.baseURL = "http://localhost:8080";
const hat = await MakeHat({ inches: 12 }); // We can omit `baseURL` because it has already been set
console.log(hat);
You can override a globally configured baseURL
at the RPC call site:
import { client } from "twirpscript";
client.baseURL = "http://localhost:8080";
// This RPC will make a request to https://api.example.com instead of http://localhost:8080
const hat = await MakeHat(
{ inches: 12 },
{ baseURL: "https://api.example.com" }
);
console.log(hat);
// This RPC will make a request to http://localhost:8080
const otherHat = await MakeHat({ inches: 12 });
console.log(otherHat);
In addition to baseUrl
, headers
can also be set at via global configuration or call site configuration. headers
defines key value pairs that become HTTP headers for the RPC:
import { client } from "twirpscript";
client.baseURL = "http://localhost:8080";
// setting a (non standard) HTTP "device-id" header via global configuration. This header will be sent for every RPC.
client.headers = { "device-id": getOrGenerateDeviceId() };
// setting a (non standard) HTTP "idempotency-key" header for this RPC call. This header will only be sent for this RPC.
const hat = await MakeHat(
{ inches: 12 },
{ headers: { "idempotency-key": "foo" } }
);
console.log(hat);
Twirp abstracts many network details from clients. Sometimes you will want more control over the underlying network request. For these cases, TwirpScript exposes rpcTransport
.
rpcTransport
can be used to customize the fetch
request made. For example, setting fetch
's credentials option to include (which will include cookies in cross origin requests). rpcTransport
can be set via global configuration or call site configuration:
import { client } from "twirpscript";
client.rpcTransport = (url, opts) =>
fetch(url, { ...opts, credentials: "include" });
// setting a custom rpcTransport for this RPC call. This transport will only be used for this RPC.
const hat = await MakeHat(
{ inches: 12 },
{
rpcTransport: (url, opts) =>
fetch(url, { ...opts, credentials: "include" }),
}
);
rpcTransport
could also be used to replace fetch
use entirely with something like axios
or an https agent.
Servers can be configured by passing a configuration object to createTwirpServer
.
Name | Description | Type | Example |
---|---|---|---|
prefix | A path prefix such as "/my/custom/prefix". Defaults to "/twirp", but can be set to "". | string | "/my/custom/prefix" |
debug | Puts the Twirp server runtime into debug mode when set to true. This enables request logging. Defaults to true. | boolean | false |
import { createServer } from "http";
import { createTwirpServer } from "twirpscript";
import { haberdasherHandler } from "./haberdasher";
const PORT = 8080;
// This removes the "/twirp" prefix in the RPC path
const app = createTwirpServer([haberdasherHandler], { prefix: "" });
createServer(app).listen(PORT, () =>
console.log(`Server listening on port ${PORT}`)
);
Name | Description |
---|---|
url | The URL for the RPC. This is the full URL for the request: the baseURL + prefix + the service path. |
headers | HTTP headers to include in the RPC. |
Name | Description |
---|---|
service | The requested RPC service. |
method | The requested RPC service method. |
contentType | The requested content-type for the request. |
Your service handlers are invoked with context
as their second argument. The base fields are documented above, but you may extend this object with arbitrary fields. This means you can use context
to provide information to your handler that doesn't come from the RPC request itself, such as http headers or server-side API invocations.
Custom fields can be added to the context object via middleware.
If you setup middleware similiar to the authentication middleware example, you could read the currentUser
username
property in your service handler. See the authentication example for a full application.
import { Haberdasher, createHaberdasher } from "../../protos/haberdasher.pb";
import { Context } from "../some-path-to-your-definition";
const haberdasher: Haberdasher<Context> = {
MakeHat: (size, ctx) => {
return {
inches: size.inches,
color: "red",
name: `${ctx.currentUser.username}'s fedora`,
};
},
};
export const haberdasherHandler = createHaberdasher(haberdasher);
TwirpScript's client and server request response lifecycle can be programmed via middleware.
Middleware is called in order of registration, with the Twirp RPC invoked last.
Because each middleware is responsible for invoking the next handler, middleware can do things like short circuit and return a response before the RPC is made, or inspect the returned response, enabling powerful patterns such as caching.
Clients can be configured via the client
export's use
method. use
registers middleware to manipulate the client request / response lifecycle. The middleware handler will receive context and next
parameters. You can set the headers and url for the RPC via context
. next
invokes the next handler in the chain -- either the next registered middleware, or the Twirp RPC.
import { client } from "twirpscript";
client.use((context, next) => {
const auth = localStorage.getItem("auth");
if (auth) {
context.headers["authorization"] = `bearer ${auth}`;
}
return next(context);
});
Client middleware can override both global configuration and call site configuration.
import { client } from "twirpscript";
client.baseURL = "http://localhost:8080";
client.use((context, next) => {
const url = new URL(context.url);
url.host = "www.foo.com";
return next({ ...context, url: url.toString() });
});
// This will make a request to https://www.foo.com instead of http://localhost:8080 or https://api.example.com"
const hat = await MakeHat(
{ inches: 12 },
{ baseURL: "https://api.example.com" }
);
console.log(hat);
Servers can be configured via your server
's use
method. use
registers middleware to manipulate the server request / response lifecycle.
The middleware handler will receive req
, context
and next
parameters. req
is the incoming request. context
is the context which will be passed to each middleware handler and finally the Twirp service handler you implemented. You may extend context
to pass extra parameters to your service handlers that are not available via your service's defined request parameters. This can be used to implement things such as authentication or rate limiting. next
invokes the next handler in the chain -- either the next registered middleware, or if there is no middleware remaining, the Twirp service handler you implemented.
import { createServer } from "http";
import { createTwirpServer, TwirpError } from "twirpscript";
import { authenticationHandler } from "./authentication";
export interface Context {
currentUser: { username: string };
}
const services = [authenticationHandler]
const app = createTwirpServer<Context, typeof services>(services);
app.use(async (req, ctx, next) => {
// exception so unauthenticated users can authenticate
if (ctx.service.name === authenticationHandler.name) {
return next();
}
// extract token from authorization header
const token = req.headers["authorization"]?.split("bearer")?.[1]?.trim();
// a fictional helper function that retrieves a user from a token
const currentUser = getCurrentUser(token);
if (!currentUser) {
return TwirpErrorResponse({
code: "unauthenticated",
msg: "Access denied",
});
} else {
ctx.currentUser = currentUser;
return next();
}
};
createServer(app).listen(PORT, () =>
console.log(`Server listening on port ${PORT}`)
);
See the authentication example for a full application.
Twirp defines a list of error codes. You can throw an error from your service handler by using TwirpError
:
import { TwirpError } from "twirpscript";
if (!currentUser) {
throw new TwirpError({
code: "unauthenticated",
msg: "Access denied",
});
}
Note: You must use TwirpError
. Any unhandled errors will otherwise be caught and the TwirpScript server will respond with the following JSON response: { code: "internal", msg: "server error" }
and set the appropriate headers and status code.
If you want to respond with a Twirp Error from middleware
, use TwirpErrorResponse
. This will create a Twirp Error response while still running any remaining middleware
in the chain. You may explictly define try / catch
clauses in your middleware, but any unhandled errors will otherwise be caught and the TwirpScript server will respond with the error described above.
TwirpScript clients and servers can be instrumented by listening to events at key moments during the request / response life cycle. These hooks are ideal placements for instrumentation such as metrics reporting and logging. Event handlers for both clients and servers are passed a context
object as the first argument. As a best practice, this context
object should be treated as readonly / immutable.
While hooks and middleware can be used to accomplish similar goals, as a best practice use hooks for instrumentation and middleware for mutation.
Every client event handler is invoked with the request context.
requestPrepared
is called as soon as a request has been created and before
it has been sent to the Twirp server.
responseReceived
is called after a request has finished sending.
error
is called when an error occurs during the sending or receiving of a
request. In addition to the context
, the error that occurred is passed as the second argument.
import { client } from "twirpscript";
client.on("responseReceived", (context) => {
// log or report
});
Every server event handler is invoked with the request context.
requestReceived
is called as soon as a request enters the Twirp
server at the earliest available moment. Called with the current context
and the request.
requestRouted
is called when a request has been routed to a
service method. Called with the current context
and the input to the service method.
responsePrepared
is called when a request has been handled by a service method. Called with the current context
and the response generated by the service method.
responseSent
is called when all bytes of a response (including an error
response) have been written. Called with the current context
and the response.
error
is called when an error occurs while handling a request. Called with the current context
and the error that occurred.
import { createServer } from "http";
import { createTwirpServer } from "twirpscript";
import { habderdasherHandler } from "./haberdasher";
const PORT = 8080;
const app = createTwirpServer([habderdasherHandler]);
app.on("responseSent", (ctx) => {
// log or report
});
createServer(app).listen(PORT, () =>
console.log(`Server listening on port ${PORT}`)
);
TwirpScript aims to be zero config, but can be configured by creating a .twirp.json
file in your project root.
Name | Description | Type |
---|---|---|
root |
The root directory. `.proto` files will be searched under this directory, and `proto` import paths will be resolved relative to this directory. TwirpScript will recursively search all subdirectories for `.proto` files.
Defaults to the project root. Example: If we have the following project structure:
Default: A.proto would import "src/B.proto"; Setting // .twirp.json {
"root": "src"
} A.proto would import "B.proto"; TypeScript projects will generally want to set this value to match their |
string (filepath) |
exclude |
An array of patterns that should be skipped when searching for `.proto` files.
Example: If we have the following project structure: /src /foo A.proto /bar B.proto Setting // .twirp.json {
"exclude": ["/bar/"]
} Will only process A.proto (B.proto) will be excluded from TwirpScript's code generation. |
string[] (RegExp pattern) |
dest |
The destination folder for generated files.
Defaults to colocating generated files with the corresponding If we have the following project structure:
TwirpScript will generate the following:
Setting // .twirp.json {
"dest": "out",
}
Note that the generated directory structure will mirror the Setting // .twirp.json {
"root": "src",
"dest": "out",
}
|
string (filepath) |
language |
Whether to generate JavaScript or TypeScript.
If omitted, TwirpScript will attempt to autodetect the language by looking for a |
javascript | typescript |
json |
JSON serializer options.
See https://developers.google.com/protocol-buffers/docs/proto3#json for more context. |
{ emitFieldsWithDefaultValues?: boolean, useProtoFieldName?: boolean } |
typecript |
TypeScript options.
|
{ emitDeclarationOnly?: boolean } |
TwirpScript's JSON serialization/deserialization is migrating to the proto3 specification. This is nearly complete, but still in progress.
TwirpScript will serialize JSON keys as lowerCamelCase
versions of the proto field. Per the proto3 spec, the runtime will accept both lowerCamelCase
and the original proto field name when deserializing. You can provide the json_name
field option to specify an alternate key name. When doing so, the runtime will encode JSON messages using the the json_name
as the key, and will decode JSON messages using the json_name
if present, otherwise falling back to the lowerCamelCase
name and finally to the original proto field name.
syntax = "proto3";
message Hat {
// this key will serialize as 'userID' instead of 'userId'
int64 user_id = 1; [json_name="userID"];
int64 wardrobe_id = 2;
}
The above Hat
message would serialize to the following JSON:
{ "userID": "some 64bit number", "wardrobeId": "some 64bit number" }
The documentation is a work in progress. Checkout the examples in the examples directory:
- The JavaScript fullstack shows a minimal browser client and server implementation in JavaScript.
- The TypeScript fullstack shows a minimal browser client and server implementation in TypeScript.
- The authentication example extends the fullstack example to demonstrate authentication using tokens.
- The aws lambda example shows TwirpScript running on AWS Lambda, complete with the necessary CDK to deploy a full stack (API Gateway + Lambda).
The examples also demonstrate testing using jest.
As a design goal, TwirpScript should always work with the strictest TypeScript compiler settings. If your generated TwirpScript files are failing type checking, please open an issue.
TwirpScript could opt generated files out from type checking, but it leverages the TypeScript compiler to flag version mismatches between the TwirpScript runtime and generated code.
TwirpScript does not make any guarantees for tools like linters and formatters such as prettier or eslint. It generally does not make sense to run these tools against code generation artifacts, like the .pb.ts
or .pb.js
files generated by TwirpScript. This is because:
- The permutations of preferences and plugins for these tools quickly explode beyond what is feasible for a single tool to target. There are always new tools that could be added as well.
- The code is generated automatically, and not all rules are autofixable. This means there are cases that would always require manual intervention by the user.
- Autogenerated code is readonly, and expected to be correct. Autogenerated code has a much difference maintenance cycle than code written by hand, and should generally be treated as a binary or a dependency. You don't lint your node_modules!
Here are some example snipits to opt TwirpScript generated code out of these tools:
.eslintrc.js
module.exports = {
ignorePatterns: ["*.pb.[t|j]s"],
};
TwirpScript can be used with Buf. This will bypass TwirpScript's cli runner, so all options must be passed to buf
via it's configuration yaml. .twirp.json
is bypassed, as is automatic typescript
inference.
buf.gen.yaml
version: v1
plugins:
- name: protoc-gen-twirpscript
path: ./node_modules/twirpscript/compiler.js
out: .
opt:
- language=typescript
strategy: all
See the buf example for a full example.
This library supports Protobuf3. Protobuf2 should generally work, with the notable exception of Groups which are deprecated and will not be implemented.
At this time, there are no plans to support Proto2 directly. I'd like any community that grows around TwirpScript to not be concerned with alternate patterns to support proto2
. TwirpScript's release came quite some time after the release of proto3
,and I'm not aware of any compelling reasons to use proto2
over proto3
for new applications. The protobuf compiler supports references between proto2
and proto3
files, so existing applications can port messages incrementally to proto3
that want to leverage TwirpScript.
If there is sufficient demand for proto2
support, I will reconsider this stance. Please open an issue and add the Protobuf 2
label if you have a proto2
support request.
Why use Twirp instead of GraphQL, gRPC or REST?
For multiple clients with distinct views, I would pull in GraphQL. For a single UI client I prefer the simpler architecture (and well defined optimization paths) found with a more traditional API server approach.
A REST or JSON API lacks type safety and the developer experience that comes from static typing. This can be mitigated to an extent with tools like JSON Schema, but that route requires stitching together (and maintaining) a suite of tools to achieve a similar developer experience.
gRPC is great, but has a large runtime (and corresponding complexity) that is unnecessary for some applications. Twirp offers many of the benefits of gRPC with a significantly reduced runtime. TwirpScript's developer experience is designed to be idiomatic for the JS/TS community, and TwirpScript's autogenerated clients are optimized for use in the browser.
To learn more about the motivation behind Twirp (and a comparison to REST APIs and gRPC), check out the announcement blog.
PR's and issues welcomed! For more guidance check out CONTRIBUTING.md
See the project's MIT License.