Protoc import errors
Closed this issue · 8 comments
I am failing to generate scala sources from pre-defined proto files, using Mu version: 0.18.0
The proto files are sourced from JARs published by the OpenCensus project, and my sbt file is configured as follows:
lazy val ocensusProto = "io.opencensus" % "opencensus-proto" % "0.2.0"
lazy val muFs2 = "io.higherkindness" %% "mu-rpc-fs2" % "0.18.0"
lazy val muRpcServer = "io.higherkindness" %% "mu-rpc-server" % "0.18.0"
lazy val protocol = project
.in(file("modules/protocol"))
.configure(BuildSettings.profile)
.settings(name := "apm-collector-protocol", libraryDependencies ++= Seq(muRpcServer, ocensusProto, scalaTest % Test))
.settings(
Seq(
idlType := "proto",
srcGenSerializationType := "Protobuf",
srcGenJarNames := Seq("opencensus-proto"),
srcGenTargetDir := (Compile / sourceManaged).value / "compiled_proto",
libraryDependencies ++= Seq("io.higherkindness" %% "mu-rpc-channel" % Versions.mu),
sourceGenerators in Compile += (Compile / srcGen).taskValue
)
)
I also have the idl-gen plugin configured as follows in plugins.sbt:
addSbtPlugin("io.higherkindness" % "sbt-mu-idlgen" % "0.18.0")
I have been scanning the sbt plugin code to parse protocol buffers files, and I see that it tries to resolve import statements that are then passed as arguments to protoc. However, when I compile it appears that the imports are not resolved correctly. The output is as follows:
sbt:apm-collector-protocol> compile
[info] Updating ...
[info] Done updating.
protoc-jar: protoc version: 3.6.0, detected platform: osx-x86_64 (mac os x/x86_64)
protoc-jar: embedded: bin/3.6.0/protoc-3.6.0-osx-x86_64.exe
protoc-jar: executing: [/var/folders/m3/_bxmtk_13z1fnm54cczf9w5h0000gn/T/protocjar3473633784860612324/bin/protoc.exe, --plugin=protoc-gen-proto2_to_proto3, --include_imports, --descriptor_set_out=trace_config.proto.desc, --proto_path=/Users/davidc/work/apm-collector/modules/protocol/target/scala-2.12/resource_managed/main/proto/opencensus/proto/trace/v1, trace_config.proto]
protoc-jar: protoc version: 3.6.0, detected platform: osx-x86_64 (mac os x/x86_64)
protoc-jar: embedded: bin/3.6.0/protoc-3.6.0-osx-x86_64.exe
protoc-jar: executing: [/var/folders/m3/_bxmtk_13z1fnm54cczf9w5h0000gn/T/protocjar16595508427019335140/bin/protoc.exe, --plugin=protoc-gen-proto2_to_proto3, --include_imports, --descriptor_set_out=trace.proto.desc, --proto_path=/Users/davidc/work/apm-collector/modules/protocol/target/scala-2.12/resource_managed/main/proto/opencensus/proto/trace/v1, trace.proto]
opencensus/proto/resource/v1/resource.proto: File not found.
google/protobuf/timestamp.proto: File not found.
google/protobuf/wrappers.proto: File not found.
trace.proto: Import "opencensus/proto/resource/v1/resource.proto" was not found or had errors.
trace.proto: Import "google/protobuf/timestamp.proto" was not found or had errors.
trace.proto: Import "google/protobuf/wrappers.proto" was not found or had errors.
trace.proto:173:5: "google.protobuf.Timestamp" is not defined.
trace.proto:133:3: "google.protobuf.Timestamp" is not defined.
trace.proto:144:3: "google.protobuf.Timestamp" is not defined.
trace.proto:297:3: "opencensus.proto.resource.v1.Resource" is not defined.
trace.proto:304:3: "google.protobuf.BoolValue" is not defined.
trace.proto:308:3: "google.protobuf.UInt32Value" is not defined.
[error] higherkindness.skeuomorph.ProtobufCompilationException: Protoc failed to compile protobuf file
[error] at higherkindness.skeuomorph.protobuf.ParseProto$.$anonfun$runProtoc$2(ParseProto.scala:65)
[error] at cats.MonadError.$anonfun$ensure$1(MonadError.scala:14)
[error] at cats.effect.internals.IORunLoop$.liftedTree3$1(IORunLoop.scala:225)
[error] at cats.effect.internals.IORunLoop$.step(IORunLoop.scala:225)
[error] at cats.effect.IO.unsafeRunTimed(IO.scala:321)
[error] at cats.effect.IO.unsafeRunSync(IO.scala:240)
[error] at higherkindness.mu.rpc.idlgen.proto.ProtoSrcGenerator$.generateFrom(ProtoSrcGenerator.scala:46)
[error] at higherkindness.mu.rpc.idlgen.Generator.$anonfun$generateFrom$1(Generator.scala:30)
[error] at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)
[error] at scala.collection.Iterator.foreach(Iterator.scala:937)
[error] at scala.collection.Iterator.foreach$(Iterator.scala:937)
[error] at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
[error] at scala.collection.IterableLike.foreach(IterableLike.scala:70)
[error] at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
[error] at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
[error] at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)
[error] at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)
[error] at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
[error] at higherkindness.mu.rpc.idlgen.Generator.generateFrom(Generator.scala:29)
[error] at higherkindness.mu.rpc.idlgen.Generator.generateFrom$(Generator.scala:25)
[error] at higherkindness.mu.rpc.idlgen.proto.ProtoSrcGenerator$.generateFrom(ProtoSrcGenerator.scala:33)
[error] at higherkindness.mu.rpc.idlgen.GeneratorApplication.generateFrom(GeneratorApplication.scala:63)
[error] at higherkindness.mu.rpc.idlgen.IdlGenPlugin$.$anonfun$idlGenTask$1(IdlGenPlugin.scala:186)
[error] at sbt.util.FileFunction$.$anonfun$cached$1(FileFunction.scala:73)
[error] at sbt.util.FileFunction$.$anonfun$cached$4(FileFunction.scala:147)
[error] at sbt.util.Difference.apply(Tracked.scala:313)
[error] at sbt.util.Difference.apply(Tracked.scala:293)
[error] at sbt.util.FileFunction$.$anonfun$cached$3(FileFunction.scala:143)
[error] at sbt.util.Difference.apply(Tracked.scala:313)
[error] at sbt.util.Difference.apply(Tracked.scala:288)
[error] at sbt.util.FileFunction$.$anonfun$cached$2(FileFunction.scala:142)
[error] at higherkindness.mu.rpc.idlgen.IdlGenPlugin$.$anonfun$taskSettings$6(IdlGenPlugin.scala:159)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:44)
[error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:40)
[error] at sbt.std.Transform$$anon$4.work(System.scala:67)
[error] at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error] at sbt.Execute.work(Execute.scala:278)
[error] at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
[error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[error] at java.base/java.lang.Thread.run(Thread.java:835)
[error] (srcGen) higherkindness.skeuomorph.ProtobufCompilationException: Protoc failed to compile protobuf file
[error] Total time: 2 s, completed 28 May 2019, 15:06:22
sbt:apm-collector-protocol>
Is there a way that I can specify dependency ordering, and also, what is the best way to include the googlle protocol buffer timestamp dependency? I have tried including the google protobuf project as a JAR to srcGen, but it also fails to compile due to import issues.
I would greatly appreciate any input as to how I can get this to compile, I am willing to create a demo project to reproduce if necessary.
Thank you for your help.
@david-cromberge The demo project would be really helpful if possible :)
Thanks!
@juanpedromoreno I have set up a demo project here:
https://github.com/david-cromberge/mu-opencensus
I have been reading the mu IDL-generation documentation. My understanding is that it should be possible to use an external JAR artifact containing proto services, to generate mu service definitions, together with message definitions, as I have attempted in the demo project above.
If I have misunderstood the documentation, or am doing something wrong, I would be glad to do anything I can to help address the problem.
Thanks.
Great, thanks @david-cromberge
Yes, your understanding is correct, you should be able to use external JARs, containing proto definitions. Let me take a look at it.
@david-cromberge
If you add the next statement to your build.sbt file:
srcGenSourceDirs ++= Seq((Compile / resourceManaged).value / "proto"),That specific problem should be fixed since now the proto files are found. However, it looks like that after adding that setting the unpackaged .proto files are overridden somehow, and all of them turn out empty. I'll keep digging into this bug.
@juanpedromoreno thank you, that is progress 👍 I have tried deepening the path, to try and constrain the output for investigation:
srcGenSourceDirs ++= Seq((Compile / resourceManaged).value / "proto" / "resource")
This mirrors the path:
https://github.com/census-instrumentation/opencensus-proto/tree/master/src/opencensus/proto/resource/v1
Unfortunately I was not able to produce valid source files, and the sources produced by the idl-gen plugin also came up empty. I will also keep looking and will inform you if anything new comes to light.
This bug is related to higherkindness/skeuomorph#111, the proposed solution is #627.
Also, while testing this bug with https://github.com/david-cromberge/mu-opencensus I discovered a new bug in skeuomorph that needs to be fixed in order to be able to completely compile that project: higherkindness/skeuomorph#114
This issue should be fixed in the latest version (0.20.1) of mu-scala. I've confirmed that the mu-opencensus repo kindly provided by @david-cromberge now compiles successfully.
As an example of a complex real-world protobuf defintion, that repo uncovered quite a few different bugs in our codegen feature, so it's been really helpful. Thank you!