trouble building under java8 on arm64
strophy opened this issue · 5 comments
Hello, I am trying to native build ScalaPB so it can be included in the rvolosatovs/docker-protobuf plugin collection, but I am having difficulty building for ARM architecture. I followed the example in the CI from this repo, which uses jabba under the hood to install graalvm@21.1.0. This resolves to graalvm-ce-java8-linux-amd64-21.1.0.tar.gz. However, under ARM only the following two packages are available in jabba:
$ uname -m
aarch64
$ jabba ls-remote | grep graalvm
graalvm-ce-java11@21.1.0
graalvm-ce-java16@21.1.0
It turns out GraalVM 21 does not support Java 8 on ARM: https://github.com/graalvm/graalvm-ce-builds/releases/tag/vm-21.1.0
The same problem means using the sbtscala/scala-sbt Docker images is also not possible.
Unfortunately attempting to build with any other version of Java fails as follows (e.g. Java 11 on x86):
#0 178.7 Downloading: Component catalog from www.graalvm.org
#0 179.4 Processing Component: Native Image
#0 179.4 Downloading: Component native-image: Native Image from github.com
#0 181.0 Installing new component: Native Image (org.graalvm.native-image, version 20.2.0)
#0 185.6 [info] /root/.cache/coursier/jvm/graalvm-java11@20.2.0/bin/native-image -cp /scala-protobuf/protoc-gen-scala-native-image/target/native-image-internal/manifest.jar -H:ReflectionConfigurationFiles=/scala-protobuf/protoc-gen-scala-native-image/native-image-config/reflect-config.json -H:Name=protoc-gen-scala --static --no-fallback scalapb.ScalaPbCodeGenerator /scala-protobuf/target/protoc-gen-scala
#0 189.4 [/scala-protobuf/target/protoc-gen-scala:498] classlist: 3,350.61 ms, 0.96 GB
#0 190.4 [/scala-protobuf/target/protoc-gen-scala:498] (cap): 598.99 ms, 0.96 GB
#0 190.5 [/scala-protobuf/target/protoc-gen-scala:498] setup: 1,137.79 ms, 0.96 GB
#0 190.5 Error: Error parsing reflection configuration in /scala-protobuf/protoc-gen-scala-native-image/native-image-config/reflect-config.json:
#0 190.5 Unknown attribute 'queriedMethods' (supported attributes: allDeclaredConstructors, allPublicConstructors, allDeclaredMethods, allPublicMethods, allDeclaredFields, allPublicFields, methods, fields) in defintion of class com.google.protobuf.DescriptorProtos$DescriptorProto
#0 190.5 Verify that the configuration matches the schema described in the -H:PrintFlags=+ output for option ReflectionConfigurationFiles.
#0 190.5 Error: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception
#0 190.6 Error: Image build request failed with exit status 1
#0 190.6 [error] native-image command failed with exit code '1'
#0 190.6 [error] (protocGenScalaNativeImage / nativeImage) native-image command failed with exit code '1'
#0 190.6 [error] Total time: 181 s (03:01), completed Jan 2, 2023, 4:48:25 AM
------
Dockerfile:335
--------------------
333 | RUN gu install native-image
334 | RUN ./make_reflect_config.sh
335 | >>> RUN sbt protocGenScalaNativeImage/nativeImage
336 | RUN install -D /scala-protobuf/target/protoc-gen-scala /out/usr/bin/protoc-gen-scala
337 |
--------------------
I don't know anything about Java or Scala toolchains, can anyone help me out here? Do I have to use GraalVM, or is some other Java base image like Zulu possible? Otherwise, if the build only runs on Java 8 and Java 8 does not support ARM, how can I run a build?
Hi @strophy
ScalaPB works on all JVMs>=8, so using a JDK that supports Java 11 should be fine. Can you confirm that the make_reflect.config.sh step generates a new reflect-config.json ? Does it have a queriedMethods key in it? If so, why graalvm doesn't recognize it? This sounds more like a question for the GraalVM team.
Yes, running make_reflect.config.sh generates protoc-gen-scala-nativ-image/native-image-config/reflect-config.json, is this the file you are expecting to see? The contents of the relevant section appear as follows:
{
"name": "com.google.protobuf.DescriptorProtos$DescriptorProto",
"methods": [
{
"name": "newBuilder",
"parameterTypes": []
}
],
"queriedMethods": [
{
"name": "getEnumType",
"parameterTypes": [
"int"
]
},
{
"name": "getEnumTypeCount",
"parameterTypes": []
},
{
"name": "getEnumTypeList",
"parameterTypes": []
},
{
"name": "getExtension",
"parameterTypes": [
"int"
]
},
{
"name": "getExtensionCount",
"parameterTypes": []
},
{
"name": "getExtensionList",
"parameterTypes": []
},
{
"name": "getExtensionRange",
"parameterTypes": [
"int"
]
},
{
"name": "getExtensionRangeCount",
"parameterTypes": []
},
{
"name": "getExtensionRangeList",
"parameterTypes": []
},
{
"name": "getField",
"parameterTypes": [
"int"
]
},
{
"name": "getFieldCount",
"parameterTypes": []
},
{
"name": "getFieldList",
"parameterTypes": []
},
{
"name": "getName",
"parameterTypes": []
},
{
"name": "getNameBytes",
"parameterTypes": []
},
{
"name": "getNestedType",
"parameterTypes": [
"int"
]
},
{
"name": "getNestedTypeCount",
"parameterTypes": []
},
{
"name": "getNestedTypeList",
"parameterTypes": []
},
{
"name": "getOneofDecl",
"parameterTypes": [
"int"
]
},
{
"name": "getOneofDeclCount",
"parameterTypes": []
},
{
"name": "getOneofDeclList",
"parameterTypes": []
},
{
"name": "getOptions",
"parameterTypes": []
},
{
"name": "getReservedName",
"parameterTypes": [
"int"
]
},
{
"name": "getReservedNameCount",
"parameterTypes": []
},
{
"name": "getReservedNameList",
"parameterTypes": []
},
{
"name": "getReservedRange",
"parameterTypes": [
"int"
]
},
{
"name": "getReservedRangeCount",
"parameterTypes": []
},
{
"name": "getReservedRangeList",
"parameterTypes": []
},
{
"name": "hasName",
"parameterTypes": []
},
{
"name": "hasOptions",
"parameterTypes": []
}
]
},
This can be reproduced quite easily using the following short Dockerfile:
FROM sbtscala/scala-sbt:${SCALA_SBT_VERSION} as protoc_gen_scala
RUN mkdir -p /scala-protobuf
RUN curl -sSL https://api.github.com/repos/scalapb/ScalaPB/tarball/v0.11.12 | tar xz --strip 1 -C /scala-protobuf
WORKDIR /scala-protobuf
RUN gu install native-image
RUN ./make_reflect_config.sh
RUN sbt protocGenScalaNativeImage/nativeImage
RUN install -D /scala-protobuf/target/protoc-gen-scala /out/usr/bin/protoc-gen-scala
Running this file with different vars for SCALA_SBT_VERSION produces the following results, just under x86_64 for now to try and get native build working under Java >8:
SCALA_SBT_VERSION=graalvm-ce-21.2.0-java8_1.8.0_2.12.17 //works
SCALA_SBT_VERSION=graalvm-ce-21.3.0-java11_1.8.0_2.12.17 //fails
SCALA_SBT_VERSION=graalvm-ce-21.3.0-java17_1.8.0_2.12.17 //fails
SCALA_SBT_VERSION=graalvm-ce-21.3.0-java17_1.8.0_3.2.1 //fails
Can you reproduce this on x86? Or should I ask GraalVM for help?
When running the Dockerfile you provided I noticed that the sbt protocGenScalaNativeImage/nativeImage step ends up downloading an older version of native-image (20.2.0), while make_reflect_config is using a more recent version. Since we install native-image manually using gu install, we need to disable the auto-download feature of sbt's native image plugin. The following dockerfile worked for me on x86 (the fix is the in ENV line):
FROM sbtscala/scala-sbt:graalvm-ce-21.3.0-java11_1.8.0_2.12.17
RUN mkdir -p /scala-protobuf
RUN curl -sSL https://api.github.com/repos/scalapb/ScalaPB/tarball/v0.11.12 | tar xz --strip 1 -C /scala-protobuf
WORKDIR /scala-protobuf
RUN gu install native-image
RUN ./make_reflect_config.sh
ENV NATIVE_IMAGE_INSTALLED=true
RUN sbt protocGenScalaNativeImage/nativeImage
RUN install -D /scala-protobuf/target/protoc-gen-scala /out/usr/bin/protoc-gen-scala
Thanks, this works perfectly now, including under ARM. You can track inclusion of the Scala plugin in rvolosatovs/protoc here: rvolosatovs/docker-protobuf#121