Gunk

Gunk is a modern frontend and syntax for Protocol Buffers.

Overview

Gunk provides a modern project-based workflow along with a Go-derived syntax for defining types and services for use with Protocol Buffers. Gunk is designed to integrate cleanly with existing protoc based build pipelines, while standardizing workflows in a way that is familiar/accessible to Go developers.

Quickstart

Create a working directory for a project:

$ mkdir -p ~/src/example && cd ~/src/example

Install gunk and place the following Gunk definitions in example/util.gunk:

package util

// Util is a utility service.
type Util interface {
	// Echo returns the passed message.
	Echo(Message) Message
}

// Message contains an echo message.
type Message struct {
	// Msg is a message from a client.
	Msg string `pb:"1"`
}

Create the corresponding project configuration in example/.gunkconfig:

[generate go]

[generate js]
import_style=commonjs
binary

Then, generate protocol buffer definitions/code:

$ ls -A
.gunkconfig  util.gunk

$ gunk generate

$ ls -A
all.pb.go  all_pb.js  .gunkconfig  util.gunk

As seen above, gunk generated the corresponding Go and JavaScript protobuf code using the options defined in the .gunkconfig.

End-to-end Example

A end-to-end example gRPC server implementation, using Gunk definitions is available for review.

Debugging `protoc` commands

Underlying commands executed by gunk can be viewed with the following:

$ gunk generate -x
protoc-gen-go
protoc --js_out=import_style=commonjs,binary:/home/user/example --descriptor_set_in=/dev/stdin all.proto

Installing

The gunk command-line tool can be installed via Release, via Homebrew, via Scoop or via Go:

Installing via Release

Download a release for your platform
Extract the gunk or gunk.exe file from the .tar.bz2 or .zip file
Move the extracted executable to somewhere on your $PATH (Linux/macOS) or %PATH% (Windows)

Installing via Homebrew (macOS)

gunk is available in the gunk/gunk tap, and can be installed in the usual way with the brew command:

# add tap
$ brew tap gunk/gunk

# install gunk
$ brew install gunk

Installing via Scoop (Windows)

gunk can be installed using Scoop:

# install scoop if not already installed
iex (new-object net.webclient).downloadstring('https://get.scoop.sh')

scoop install gunk

Installing via Go

gunk can be installed in the usual Go fashion:

# install gunk
$ go get -u github.com/gunk/gunk

Protobuf Dependency and Caching

The gunk command-line tool uses the protoc command-line tool. gunk can be configured to use protoc at a specified path. If it isn't available, gunk will download the latest protobuf release to the user's cache, for use. It's also possible to pin a specific version, see the section on protoc configuration.

Protocol Types and Messages

Gunk provides an alternate, Go-derived syntax for defining protocol buffers. As such, Gunk definitions are a subset of the Go programming language. Additionally, a special +gunk annotation is recognized by gunk, to allow the declaration of protocol buffer options:

package message

import "github.com/gunk/opt/http"

// Message is a Echo message.
type Message struct {
	// Msg holds a message.
	Msg  string `pb:"1" json:"msg"`
	Code int    `pb:"2" json:"code"`
}

// Util is a utility service.
type Util interface {
	// Echo echoes a message.
	//
	// +gunk http.Match{
	// 	Method:	"POST",
	// 	Path:	"/v1/echo",
	// 	Body:	"*",
	// }
	Echo(Message) Message
}

Technically speaking, gunk is not actually strict subset of go, as gunk allows unused imports; it actually requires them for some features.

See the example above; in pure go, this would not be a valid go code, as http is not used outside of the comment.

Scalars

Gunk's Go-derived syntax uses the canonical Go scalar types of the proto3 syntax, defined by the protocol buffer project:

Proto3 Type	Gunk Type
`double`	`float64`
`float`	`float32`
`int32`	`int`
`int32`	`int32`
`int64`	`int64`
`uint32`	`uint`
`uint32`	`uint32`
`uint64`	`uint64`
`bool`	`bool`
`string`	`string`
`bytes`	`[]byte`

Note: Variable-length scalars will be enabled in the future using a tag parameter.

Messages

Gunk's Go-derived syntax uses Go's struct type declarations for declaring messages, and require a pb:"<field_number>" tag to indicate the field number:

type Message struct {
	FieldA string `pb:"1"`
}

type Envelope struct {
	Message Message  `pb:"1" json:"msg"`
}

There are additional tags (for example, the json: tag above), that will be recognized by gunk format, and passed on to generators, where possible.

Note: When using gunk format, a valid pb:"<field_number>" tag will be automatically inserted if not declared.

Services

Gunk's Go-derived syntax uses Go's interface syntax for declaring services:

type SearchService interface {
	Search(SearchRequest) SearchResponse
}

The above is equivalent to the following protobuf syntax:

service SearchService {
  rpc Search (SearchRequest) returns (SearchResponse);
}

Enums

Gunk's Go-derived syntax uses Go const's for declaring enums:

type MyEnum int

const (
	MYENUM MyEnum = iota
	MYENUM2
)

Note: values can also be fixed numeric values or a calculated value (using iota).

Maps

Gunk's Go-derived syntax uses Go map's for declaring map fields:

type Project struct {
	ProjectID string `pb:"1" json:"project_id"`
}

type GetProjectResponse struct {
	Projects map[string]Project `pb:"1"`
}

Repeated Values

Gunk's Go-derived syntax uses Go's slice syntax ([]) for declaring a repeated field:

type MyMessage struct {
	FieldA []string `pb:"1"`
}

Message Streams

Gunk's Go-derived syntax uses Go chan syntax for declaring streams:

type MessageService interface {
	List(chan Message) chan Message
}

The above is equivalent to the following protobuf syntax:

service MessageService {
  rpc List(stream Message) returns (stream Message);
}

Protocol Options

Protocol buffer options are standard messages (ie, a struct), and can be attached to any service, message, enum, or other other type declaration in a Gunk file via the doccomment preceding the type, field, or service:

// MyOption is an option.
type MyOption struct {
	Name string `pb:"1"`
}

// +gunk MyOption {
// 	Name: "test",
// }
type MyMessage struct {
	/* ... */
}

Project Configuration Files

Gunk uses a top-level .gunkconfig configuration file for managing the Gunk protocol definitons for a project:

# Example .gunkconfig for Go, grpc-gateway, Python and JS
[generate go]
out=v1/go
plugins=grpc

[generate]
out=v1/go
command=protoc-gen-grpc-gateway
logtostderr=true

[generate python]
out=v1/python

[generate js]
out=v1/js
import_style=commonjs
binary

Project Search Path

When gunk is invoked from the command-line, it searches the passed package spec (or current working directory) for a .gunkconfig file, and walks up the directory hierarchy until a .gunkconfig is found, or the project's root is encountered. The project root is defined as the top-most directory containing a .git subdirectory, or where a go.mod file is located.

Format

The .gunkconfig file format is compatible with Git config syntax, and in turn is compatible with the INI file format:

[generate]
command=protoc-gen-go

[generate]
out=v1/js
protoc=js

Global section

import_path - see "Converting Existing Protobuf Files"
strip_enum_type_names - with this option on, enums with their type prefixed will be renamed to the version without prefix.

Note that this might produce invalid protobuf that stops compiling in 1.4.* protoc-gen-go, if the enum names clash.

Section `[protoc]`

The path where to check for (or where to download) the protoc binary can be configured. The version can also be pinned.

Parameters

version - the version of protoc to use. If unspecified, defaults to the latest release available. Otherwise, gunk will either download the specified version, or check that the version of protoc at the specified path matches what was configured.
path - the path to check for the protoc binary. If unspecified, defaults appropriate user cache directory for the user's OS. If no file exists at the path, gunk will attempt to download protoc.

Section `[generate[ <type>]]`

Each [generate] or [generate <type>] section in a .gunkconfig corresponds to a invocation of the protoc-gen-<type> tool.

Parameters

Each name[=value] parameter defined within a [generate] section will be passed as a parameter to the protoc-gen-<type> tool, with the exception of the following special parameters that override the behavior of the gunk generate tool:

command - overrides the protoc-gen-* command executable used by gunk generate. The executable must be findable on $PATH (Linux/macOS) or %PATH% (Windows), or may be the full path to the executable. If not defined, then command will be protoc-gen-<type>, when <type> is the value in [generate <type>].
protoc - overrides the <type> value, causing gunk generate to use the protoc value in place of <type>.
out - overrides the output path of protoc. If not defined, output will be the same directory as the location of the .gunk files.
plugin_version - specify version of plugin. The plugin is downloaded from github/maven, built in cache and used. It is not installed in $PATH. This currently works with the following plugins:
- protoc-gen-go
- protoc-gen-grpc-java
- protoc-gen-grpc-gateway
- protoc-gen-openapiv2 (protoc-gen-swagger support is deprecated)
- protoc-gen-swift (installing swift itself first is necessary)
- protoc-gen-grpc-swift (installing swift itself first is necessary)
- protoc-gen-ts (installing node and npm first is necessary)
- protoc-gen-grpc-python (cmake, gcc is necessary; takes ~10 minutes to clone build)
It is recommended to use this function everywhere, for reproducible builds, together with version for protoc.
json_tag_postproc - uses json tags defined in gunk file also for go-generated file
fix_paths_postproc - for js and ts - by default, gunk generates wrong paths for other imported gunk packages, because of the way gunk moves files around. Works only if js also has import_style=commonjs option.

All other name[=value] pairs specified within the generate section will be passed as plugin parameters to protoc and the protoc-gen-<type> generators.

Short Form

The following .gunkconfig:

[generate go]

[generate js]
out=v1/js

is equivalent to:

[generate]
command=protoc-gen-go

[generate]
out=v1/js
protoc=js

Different forms of invocation

There are three different forms of gunkconfig sections that have three different semantics.

[generate]
command=protoc-gen-go

[generate]
protoc=go

[generate go]

The first one uses protoc-gen-go plugin directly, without using protoc. It also attempts to move files to the same directory as the gunk file.

The second one uses protoc and does not attempt to move any files. Protoc attempts to load plugin from $PATH, if it is not one of the built-in protoc plugins; this will not work together with pinned version and other gunk features and is not recommended outside of built-in protoc generators.

The third version is reccomended. It will try to detect whether language is one of built-in protoc generators, in that case behaves like the second way, otherwise behaves like the first.

The built-in protoc generators are:

cpp
java
python
php
ruby
csharp
objc
js

Third-Party Protobuf Options

Gunk provides the +gunk annotation syntax for declaring protobuf options, and specially recognizes some third-party API annotations, such as Google HTTP options, including all builtin/standard protoc options for code generation:

// +gunk java.Package("com.example.message")
// +gunk java.MultipleFiles(true)
package message

import (
	"github.com/gunk/opt/http"
	"github.com/gunk/opt/file/java"
)

type Util interface {
	// +gunk http.Match{
	// 	Method:	"POST",
	// 	Path:	"/v1/echo",
	// 	Body:	"*",
	// }
	Echo()
}

Further documentation on available options can be found at the Gunk options project.

Formatting Gunk Files

Gunk provides the gunk format command to format .gunk files (akin to gofmt):

$ gunk format /path/to/file.gunk
$ gunk format <pathspec>

Converting Existing Protobuf Files

Gunk provides the gunk convert command that will converting existing .proto files (or a directory) to the Go-derived Gunk syntax:

$ gunk convert /path/to/file.proto
$ gunk convert /path/to/protobuf/directory

If your .proto is referencing another .proto from another directory, you can add import_path in the global section of your .gunkconfig. If you don't provide import_path it will only search in the root directory.

import_path=relative/path/to/protobuf/directory

The path to provide is relative from the .gunkconfig location.

Furthermore, the referenced files must contain:

option go_package="path/of/go/package";

The resulting .gunk file will contain the import path as defined in go_package:

import (
	name "path/of/go/package"
)

About

Gunk is developed by the team at Brankas, and was designed to streamline API design and development.

History

From the beginning of the company, the Brankas team defined API types and services in .proto files, leveraging ad-hoc Makefile's, shell scripts, and other non-standardized mechanisms for generating Protocol Buffer code.

As development exploded in 2017 (and beyond) with continued addition of backend microservices/APIs, more code repositories and projects, and team members, it became necessary to standardize tooling for the organization as well as reduce the cognitive load of developers (who for the most part were working almost exclusively with Go) when declaring gRPC and REST services.

Naming

The Gunk name has a cheeky, backronym "Gunk Unified N-terface Kompiler", however the name was chosen because it was possible to secure the GitHub gunk project name, was short, concise, and not used by other projects.

Additionally, "gunk" is an apt description for the "gunk" surrounding protocol definition, generation, compilation, and delivery.

Contributing

Issues, Pull Requests, and other contributions are greatly welcomed and appreciated! Get started with building and running gunk:

# clone source repository
$ git clone https://github.com/gunk/gunk.git && cd gunk

# force GO111MODULES
$ export GO111MODULE=on

# build and run
$ go build && ./gunk

Dependency Management

Gunk uses Go modules for dependency management, and as such requires Go 1.11+. Please run go mod tidy before submitting any PRs:

$ export GO111MODULE=on
$ cd gunk && go mod tidy

hhhapz/gunk

Gunk

Overview

Quickstart

End-to-end Example

Debugging protoc commands

Installing

Installing via Release

Installing via Homebrew (macOS)

Installing via Scoop (Windows)

Installing via Go

Protobuf Dependency and Caching

Protocol Types and Messages

Scalars

Messages

Services

Enums

Maps

Repeated Values

Message Streams

Protocol Options

Project Configuration Files

Project Search Path

Format

Global section

Section [protoc]

Parameters

Section [generate[ <type>]]

Parameters

Short Form

Different forms of invocation

Third-Party Protobuf Options

Formatting Gunk Files

Converting Existing Protobuf Files

About

History

Naming

Contributing

Dependency Management

Debugging `protoc` commands

Section `[protoc]`

Section `[generate[ <type>]]`