Serialize and deserialize protocol buffers
kyleconroy opened this issue ยท 12 comments
As outlined in #447, TinyGo should support common serialization formats. While most people may think of JSON, I personally want TinyGo to support protocol buffers. This issue is meant to track my progress on adding Protocol Buffer support to TinyGo.
Design
The default Protocol Buffer package (https://pkg.go.dev/google.golang.org/protobuf) makes heavy use of reflection. TinyGo only supports a small amount of the reflect
package. Heavy use of reflection leads to slow code, so the folks at PlanetScale wrote vtprotobuf, which generates explicit serialization / deserialization code. This code works today with TinyGo. However, it requires the existing, slow code to compile.
So the plan is simple: Support compiling the code generated by protoc-gen-go and running and of the code contained in init()
functions. Use the code generated by vtprotobuf to serialize and deserialize the data.
Roadmap
- Compile code generated by https://github.com/planetscale/vtprotobuf
- Compile code generated by protoc-gen-go
- Import code generated by protoc-gen-go
@kyleconroy This is a great idea, I'm using vtprotobuf a lot currently and it works great. Having some kind of auto-transpiler that automatically converts the reflection-based protobuf-go to use vtprotobuf would be great.
Quick question: as of March 2023, it seems like most of the reflection features are supported - is there anything missing for the existing reflection version to work currently?
And even more reflect is working, including the last bug on you roadmap.
@kyleconroy @dgryski looking at the roadmap is it time for the confetti emoji?
So it seems that the default protobuf compiler is not supported still even with the improvements to reflection. I have yet to test vtprotobuf
% ./wasmtime-v14.0.4-x86_64-linux/wasmtime run protobuf-example.wasm
panic: unimplemented: (reflect.Type).MethodByName()
Error: failed to run main module `protobuf-example.wasm`
Caused by:
0: failed to invoke command default
1: error while executing at wasm backtrace:
0: 0x5a15 - <unknown>!runtime._panic
1: 0x5939 - <unknown>!(reflect.rawType).MethodByName
2: 0x6484 - <unknown>!(*reflect.rawType).MethodByName
3: 0x103253 - <unknown>!(*google.golang.org/protobuf/internal/impl.MessageInfo).makeStructInfo
4: 0xb95e6 - <unknown>!(*google.golang.org/protobuf/internal/impl.MessageInfo).init
5: 0xb8e02 - <unknown>!google.golang.org/protobuf/proto.protoMethods
6: 0x34c67 - <unknown>!(google.golang.org/protobuf/proto.MarshalOptions).marshal
7: 0x2d45a - <unknown>!runtime.run$1
8: 0x2cf76 - <unknown>!runtime.run$1$gowrapper
9: 0xdc1 - <unknown>!tinygo_launch
10: 0x2ce48 - <unknown>!_start
note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information
2: wasm trap: wasm `unreachable` instruction executed
FWIW I have successfully used https://github.com/planetscale/vtprotobuf to marshal/unmarshal protobuf in tinygo -target=wasi
(this can be closed then?)
How are you able to use vtprotobuf with tinygo? Isn't the generated code from vtprotobuf depends on the result of protobuf-go? I am still getting the same error with my _vtproto.pb.go
file.
How are you able to use vtprotobuf with tinygo? Isn't the generated code from vtprotobuf depends on the result of protobuf-go?
The reflection methods are stubbed out, so it still compiles and the vtprotobuf version doesn't invoke the reflection methods, so it works fine for me.
I am still getting the same error with my _vtproto.pb.go file.
Post the error and make sure you're on the latest tinygo version
I am on the latest tinygo v0.31.1, and I am also using it with wasmtime. The error I got is exactly the same with the one you posted before without using vtprotobuf. Did you change any of the code generated by protoc --go_out
?
no modifications needed. Can you post a reproduction?
I am so sorry to bother u, it was me making a dumb mistake causing the error, I fixed the error and it works totally fine. Thank you so much.
vtprotobuf generates static code for Marshal and Unmarshal. We just need a version of protoc-gen-go that does not import reflect nor any of the "heavy" protobuf packages that add a lot of complexity to the resulting binary.
I have forked to protoc-gen-go-lite here and have already dropped everything but the protoc plugin, and am working on adjusting the compiler to generate bare structs without referencing reflect: aperturerobotics/protobuf-go-lite#1
Please join the effort there as this is intended to make it easier to build lightweight tinygo protobuf projects.
I have completed protobuf-go-lite which is a reflection-free fork of protobuf-go merged with vtprotobuf and protoc-gen-go-json.
Library: https://github.com/aperturerobotics/protobuf-go-lite
Example: https://github.com/aperturerobotics/protobuf-project/tree/protobuf-lite
RPCs are available as well with starpc: https://github.com/aperturerobotics/starpc
protobuf-go-lite now supports reflection-free protobuf and protojson encoding!