Support for reading proto files from jar / zip files

[tsipo]

Hi Joshua,

I was running into this using grpcurl but as I debugged grpcurl I have noticed that the errors are returned from the protoparse package in this repo.

When one is generating protobuf code for Java - I use the gradle plugin - both the generated Java code and the proto files are all assembled into a jar file (which is actually a zip file). The structure is similar to this jar file which contains the protobuf well-known types of Google. If you open it with any archive manager, you'll see that the generated and compiled Java code is under com/google/protobuf and the proto files are assembled under google/protobuf. So in a jar file built by the service provider the Java code will typically be under com/example/... and the proto files under example/... (these names are arbitrary and can be configured, of course).

However, protparse supports only proto files directly on the file system. And whereas one can add a FileAccessor (to read the proto file from a zip file), protoparse.ResolveFilenames() (which is the first line in grpcurl.DescriptorSourceFromProtoFiles()) still assumes plain filesystem location and fails (I've debugged it through).

So currently (as some gRPC servers are internal and not using reflection), we are forced to generate both the jar file AND a FileDescriptorSet file to use by grpcurl using the --protoset flag. This is not only cumbersome, it also introduces the risk of mismatch between the two. It'd be very helpful if protoparse (and accordingly grpcurl) could use proto files residing within a jar / zip file.

Thank you!

[jhump]

Protoparse is just a parser/compiler for source. It is up to the caller to retrieve the source (such as from a JAR or ZIP file).

So grpcurl could ostensibly handle this, examining the contents of ZIP files and providing them to protoparse. But I'm not sure that the maintainers would be very interested in this.

For now, you can work-around this by unzipping the file separately and then pointing grpcurl directly at the unzipped sources.

Another similar and newer tool that may work is the Buf CLI. It includes a buf curl sub-command that is similar to grpcurl. But you can direct it to look at a ZIP file or even remote git repo in the --schema argument.

FYI, if you look at the notice at the top of this repo's README, you'll see that protoparse is not going to be materially maintained going forward. In an upcoming v2 of this repo, it will be omitted. Instead see https://github.com/bufbuild/protocompile for a newer Go library that compiles Protobuf sources.

Good luck!

[tsipo]

Thanks a lot!

I recall seeing a demo of buf a while ago, which focused mainly on their BSR service. I did not know they added curl-like capabilities.

They cannot take a .jar file as schema (it looks to me that the file extension determines the handling and only ".zip" is zip), but that can be resolved easily, like with the following symlink (which is less of a maintenance pain than making sure the jar file and a FileDescriptorSet are in sync):

$ ln -s example-grpc.jar example-grpc.zip
$ buf curl --protocol grpc --http2-prior-knowledge --schema example-grpc.zip \
http://localhost:50001/example.ExampleService/ping
{"msg":"Success"}

[jhump]

@tsipo, I don't think you actually need a sym-link. I think you can suffix the file name with #format=zip. See docs.

So try this:

buf curl --protocol grpc --http2-prior-knowledge --schema example-grpc.jar#format=zip

I did not know they added curl-like capabilities.

Yeah, I did that a few months ago 😄

[tsipo]

#format=zip suffix works like magic, thanks again!