Missing documentation / example how to use protobufs in Soy
vorburger opened this issue · 5 comments
I want to use protobufs in Soy / Closure Templates, and have tried a few things, but so far couldn't get it to work.
Perhaps I could get some tips from any maintainers? I would be willing to contribute an example or documentation about this for the next user based on what I learnt.
PS: For full disclosure, I'm a Googler not using Soy at work who wants to use Soy / Closure Templates in a personal open source project https://github.com/enola-dev/enola to generate some documentation in Markdown, and then for a simple Web UI, for now server-side Java only, no JS.
This eseems to have been introduced in 92286d9 for #99 ... but the tests there seem fairly "low-level" (to me), not exaxmples/guides.
https://github.com/google/closure-templates/blob/master/documentation/reference/types.md#protos-foobarbazproto-proto documents proto support, but doesn't really say exactly how to set it up to use this. It links to https://github.com/google/closure-templates/blob/master/documentation/dev/protos.md - which 404s (there is no protos.md in documentation/dev).
I want to use protobufs in Soy / Closure Templates, and have tried a few things, but so far couldn't get it to work.
From error messages, I gathered that apparently the {@param} has to NOT use the Proto Message Descriptor's FQN, but just it's "local" name? So not e.g. {@param package: dev.enola.core.ID} (that leads to: /dev/enola/core/docgen/markdown.soy:10: error: Proto types must be imported rather than referenced by their fully qualified names.), but just e.g. {@param package: ID}. That however then causes error: Unknown type 'ID'.
I've also discovered that SoyFileSet.Builder has an addProtoDescriptors(GenericDescriptor... | Iterable<? extends GenericDescriptor> descriptors) and called that (e.g. passing ID.getDescriptor()) but that didn't help.
I wonder if Proto Message have to be imported? I've then tried adding e.g. import {ID} from 'dev/enola/core/enola_core.proto'; ... that however leads to error: Unknown import dep dev/enola/core/enola_core.proto.
Why does it need to know and import the .proto file, if it has the Descriptor? And if it does need it, how do you tell it where to import the proto from? I've tried to add() it to the SoyFileSet.Builder, but that wasn't a good idea - causing: errors during Soy compilation /dev/enola/core/enola_core.proto:17: error: parse error at 'syntax': expected eof, {namespace, or {modname - clearly add() is only for .soy not to import .proto.
https://github.com/bazelbuild/rules_closure is something I've stumbled upon (from https://groups.google.com/g/closure-templates-discuss/c/FibvOygEcmE/m/ExPyHJNmBQAJ), but don't understand yet how that may help to use protos in Soy templates on server-side Java (from what little I can tell those rules are for more interesting for JS, and for https://github.com/google/closure-templates/blob/master/documentation/codelabs/helloworlds/helloworld_java.md#using-soyparseinfogenerator which is cool but presumably won't help me here?).
@lukesandberg or @jart or @mikesamuel or @hochhaus perhaps you have a minute to chime in here?
I'm honestly not sure how often protos are used in open source... You sound close though. I think you may just need to figure out the correct import path for the proto. I believe the import path is calculated at
You could put a print statement around there to print out the path, and try including that in the import (import {ID} from 'path/to/enola_core.proto';)
Note to my future self (or anyone else who wants to help with this - perhaps @gabrielh could be interested in this; we'll talk!) about exactly how to reproduce the problem described above to debug it: Basically use the SoyGenerator by un-commenting related stuff in the MarkdownDocGenerator and then running the MarkdownDocGeneratorTest and possibly fixing that weird import in my markdown.soy...
I have started to properly debug this: The interesting function to poke around is the run() of the class ImportsPass, which is where the related magic was (not) happening... The 2 ImportProcessor being asked if they handlesPath() are the TemplateImportProcessor (ignore that) and the ProtoImportProcessor - aha! Its Map pathToDescriptor contains core/lib/src/main/java/dev/enola/core/enola_core.proto - and not dev/enola/core/enola_core.proto. So changing:
import {ID} from 'dev/enola/core/enola_core.proto';
to
import {ID} from 'core/lib/src/main/java/dev/enola/core/enola_core.proto';
seems to do the trick. But it seems weird, to me, to have a template contain this "filesystem path" instead of the "classpath" of the proto... will this still work when I "deploy" (package) my code instead of running it e.g. from a bazelisk test "in workspace"? That path "won't exist" like that anymore in a deployment.
seems weird, to me, to have a template contain this "filesystem path" instead of the "classpath" of the proto... will this still work when I "deploy" (package) my code instead of running it e.g. from a bazelisk test "in workspace"? That path "won't exist" like that anymore in a deployment.
Oh... I figured out how this is implemented; it WILL work:
Soy doesn't actually "load .proto from classpath", as I had somehow just assumed (without really verifying). Instead, when you addProtoDescriptors() your Descriptors.GenericDescriptor to the SoyFileSet.builder(), that ends up using Proto's FileDescriptor::getName (in the ProtoImportProcessor constructor where that Map pathToDescriptor is created).
That name of a .proto (= FileDescriptor) is its "(WORKSPACE) relative path" - which is written into the Proto generated .java file at build time (look for a static { java.lang.String[] descriptorData = { "\n6core/lib/src/main/java/dev/enola/core/" + ...).
Curious, but that's how Protobuf seems to work, so be it. (I'm also not sure how this can work with other build systems than BAZEL, like Maven or Gradle, which do not have the concept of a "workspace".)
enola-dev/enola#127 marks the beginning of using closure-templates in https://enola.dev!
Closing this. @emspishak thanks for the support and "sound close" encouragement above.