API Client Generator for Go

A generator for protocol buffer described APIs for and in Go.

This is a generator for API client libraries for APIs specified by protocol buffers, such as those inside Google. It takes a protocol buffer (with particular annotations) and uses it to generate a client library.

Purpose

We aim for this generator to replace the older monolithic generator. Some areas we hope to improve over the old generator are:

using explicit normalized format for specifying APIs,
simpler, faster implementation, and
better error reporting.

Installation

go get github.com/googleapis/gapic-generator-go/cmd/protoc-gen-go_gapic. If you are using Go 1.11 and see error cannot find main module, see this FAQ page.

Or to install from source:

git pull https://github.com/googleapis/gapic-generator-go.git
cd gapic-generator-go
go install ./cmd/protoc-gen-go_gapic

The generator works as a protoc plugin, get protoc from google/protobuf.

Configuration

The generator is configured via protobuf annotations found at googleapis/api-common-protos. The generator follows the guidance defined in AIP-4210.

The only required annotation to generate a client is the service annotation google.api.default_host (here).

The value of google.api.default_host must be just a host name, excluding a scheme. For example,

import "google/api/client.proto";
...

service Foo {
    option (google.api.default_host) = "api.foo.com";
    ...
}

If a RPC returns a google.longrunning.Operation, the RPC must be annotated with google.longrunning.operation_info in accordance with AIP-151.

The Cloud Natural Language API is an example of a fully configured API that has a Go client generated by gapic-generator-go.

The supported configuration annotations include:

Service Options
- google.api.default_host: host name used in the default service client initialization
- google.api.oauth_scopes: OAuth scopes needed by the client to auth'n/z
Method Options
- google.longrunning.operation_info: used to determine response & metadata types of LRO methods

Invocation

protoc -I $API_COMMON_PROTOS --go_gapic_out [OUTPUT_DIR] --go_gapic_opt 'go-gapic-package=package/path/url;name' a.proto b.proto

Note: The $API_COMMON_PROTOS variable represents a path to the googleapis/api-common-protos directory to import the configuration annotations.

The go_gapic_opt protoc plugin option flag is necessary to convey configuration information not present in the protos. The plugin option's value is a key-value pair delimited by an equal sign =. The configuration supported by the plugin option includes:

go-gapic-package: the Go package of the generated client library.
- The substring preceding the semicolon is the import path of the package, e.g. github.com/username/awesomeness.
- The substring after the semicolon is the name of the package used in the package statement.
Note: Idiomatically the name is last element of the path but it need not be. For instance, the last element of the path might be the package's version, and the package would benefit from a more descriptive name.
grpc-service-config: the path to a gRPC ServiceConfig JSON file.
- This is used for client-side retry configuration in accordance with AIP-4221
release-level: the client library release level.
- Defaults to empty, which is essentially the GA release level.
- Acceptable values are alpha and beta.
gapic-service-config: the path the service YAML file.
- This is used for service-level client documentation.
- Note: This option is a workaround and will be deprecated.
sample: path to sample configuration files.
- This is used for sample generation. Refer to sample generation guide for more details.
gapic: path to the legacy gapic configuration file.
- This is used for sample generation only. Both gapic config itself and this option will be deprecated soon. Refer to sample generation guide for more details.

Bazel

The generator can be executed via a Bazel BUILD file using the macro in this repo.

Add the following to your WORKSPACE to import this project.

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "com_googleapis_gapic_generator_go",
    strip_prefix = "gapic-generator-master",
    urls = ["https://github.com/googleapis/gapic-generator-go/archive/master.zip"],
)

load("@com_googleapis_gapic_generator_go//:repositories.bzl", "com_googleapis_gapic_generator_go_repositories")

com_googleapis_gapic_generator_go_repositories()

Note: do not use master, use a commit hash or a release tag.

And invoke it in a BUILD file like so, using an example based on the googleapis repo.

load("@com_googleapis_gapic_generator_go//:rules_go_gapic/go_gapic.bzl", "go_gapic_library")

go_gapic_library(
  name = "language_go_gapic",
  srcs = [
    # BUILD target for proto_library
    "//google/cloud/language/v1:language_proto",
  ],
  deps = [
    # BUILD target for go_library_proto
    "//google/cloud/language/v1:language_go_proto",
  ],
  # go-gapic-package parameter value
  importpath = "cloud.google.com/go/language/apiv1;language",
)

The generator options defined in Invocation are supported as the following attributes:

grpc_service_config: a label for a gRPC ServiceConfig JSON file.
release_level: the client library release level.
service_yaml: a label for a service YAML file.
- Note: This option will eventually be deprecated.

Docker Wrapper

The generator can also be executed via a Docker container. The image containes protoc, the microgenerator binary, and the standard API protos.

$ docker run \
  --rm \
  --user $UID \
  --mount type=bind,source=</abs/path/to/protos>,destination=/in,readonly \
  --mount type=bind,source=</abs/path/to/configs>,destination=/conf,readonly \
  --mount type=bind,source=$GOPATH/src,destination=/out/ \
  gcr.io/gapic-images/gapic-generator-go \
  --go-gapic-package "github.com/package/import/path;name"

Replace /abs/path/to/protos with the absolute path to the input protos and github.com/package/import/path;name with the desired import path & name for the gapic, as described in Invocation.

For convenience, the gapic.sh script wraps the above docker invocation. An equivalent invocation using gapic.sh is:

$ gapic.sh \
  --image gcr.io/gapic-images/gapic-generator-go \
  --in /abs/path/to/protos \
  --out $GOPATH/src \
  --go-gapic-package "<github.com/package/import/path;name>"

Use gapic.sh --help to print the usage documentation.

Code Generation

This is an explanation of the Go GAPIC generator for those interested in how it works and possibly those using it as a reference.

Plugin interface

gapic-generator-go is a protoc plugin. It consumes a serialzed CodeGeneratorRequest on stdin and produces a serialized CodeGeneratorResponse on stdout. The CodeGeneratorResponse contains all of the generated Go code and/or any error(s) that might of occured during generation. All logs are emitted on stderr.

The plugin implementation can be found in cmd/protoc-gen-go_gapic.

Generated Artifacts

A single invocation of the code generator creates a doc.go file package level documentation according to godoc. This documentation is (currently) pulled from a given service config.

Each service found in the input protos gets two generated artifacts:

{service}_client.go: contains the GAPIC implementation
{service}_client_example_test.go: contains example code for each service method, consumed by godoc

There is no directory structure in the generated output. All files are placed directly in the designated output directory by protoc.

Generation Process

The generator implementation can be found in internal/gengapic.

The service client type, initialization code and any standard helpers are generated first. Then each method is generated. Any relevant helper types (i.e. pagination Iterator types, LRO helpers, etc.) for the service methods are generated following the methods.

Following the client implementation, the client example file is generated, and after all services have been generated the single doc.go file is created.

Go Version Supported

The generator itself supports the latest version.

The generated code is compatible with Go 1.6.

Contributing

If you are looking to contribute to the project, please see CONTRIBUTING.md for guidelines.

vchudnov-g/gapic-generator-go