Netty Prometheus Integration
TannerYoung opened this issue · 7 comments
Following the Metrics Reporting Guide the server sets up a metric reporter using GrpcServer.default. When I attempt to use GrpcServer.netty I get the following error when pointing prometheus at the service:
INFO: Transport failed
io.netty.handler.codec.http2.Http2Exception: Unexpected HTTP/1.x request: GET /metrics my_server
at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:85)
at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:314)
at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:251)
....
Is it possible to use this metric reporter with the netty server version?
Hopefully relevant code bits:
object CombinedServerMain extends IOApp with CommonRuntime {
...
override def run(args: List[String]): IO[ExitCode] = {
for {
metricsOps <- PrometheusMetrics.build[IO](CollectorRegistry.defaultRegistry)
interceptor = MetricsServerInterceptor(metricsOps)
accountService <- AccountService.bindService[IO].map(_.interceptWith(interceptor)).map(AddService)
server <- GrpcServer.netty[IO](10123, accountService :: Nil)
runServer <- GrpcServer.server[IO](server).as(ExitCode.Success)
} yield runServer
}
When using GrpcServer.default[IO] instead of netty; I get the following exception:
java.lang.NoClassDefFoundError: io/grpc/internal/WithLogId
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.buildTransportServer(NettyServerBuilder.java:432)
at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.buildTransportServer(NettyServerBuilder.java:55)
at io.grpc.internal.AbstractServerImplBuilder.build(AbstractServerImplBuilder.java:223)
at higherkindness.mu.rpc.server.GrpcServer$.buildServer(GrpcServer.scala:113)
Is prometheus integration in a known good state and I'm doing something wrong? Is there updated documentation on how to use it? I'm dependening on the following targets:
io.higherkindness:mu-rpc-netty:0.18.4
io.higherkindness:mu-rpc-server:0.18.4
io.higherkindness:mu-rpc-prometheus:0.18.4
It looks a binary incompatibility. Could you please share the output of the sbt evicted command?
I've got this project with a sample server/client using prometheus
I'm using bazel (not sbt) so that very likely could be the issue..
Thanks for the pointer to the example; I'll take a look tomorrow.
Looks like I was transitively depending on an incompatible version of grpc_netty_shaded via firebase_admin:
com/google/firebase:firebase_admin
com/google/cloud:google_cloud_firestore
io/grpc:grpc_netty_shaded
I've now pinned grpc-netty-shaded to version to 1.18.0 which solves the NoClassDefFound problem.
I'm still experiencing the Unexpected HTTP/1.x request: GET /metrics with both nettty and default with slightly different stack traces:
default
INFO: Transport failed
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception: Unexpected HTTP/1.x request: GET /metrics
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:85)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:314)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode
...
netty
INFO: Transport failed
io.netty.handler.codec.http2.Http2Exception: Unexpected HTTP/1.x request: GET /metrics
at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:85)
at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString
...
Not sure where to go from here, I've made a small IOApp which serves a single service with a single proto endpoint. I've copied the imports from the example app you gave to make sure I wasn't missing anything, and otherwise I have just 3 dependencies beyond my proto definition:
io/higherkindness:mu_rpc_prometheus:0.18.4
io/higherkindness:mu_rpc_server:0.18.4
org/typelevel:cats_effect:2.0.0
Do I need additional dependencies to support the http requests to /metrics? It doesn't look like example project has any additional dependencies I'm missing that the server is using save slf4j and maybe fs2. But I've attempted adding those as well for a sanity check and they have no impact. I've also enabled the paradise macro compiler plugin, but unsurprisingly that didn't fix it either.
Not sure what else to try.
@TannerYoung yeah, you need to provide that HTTP endpoint. We've got an article that describes all the process: https://www.47deg.com/blog/metrics-integration-with-mu/#checking-metrics-through-prometheus-server-11
Basically, you need to start a server that uses the CollectorRegistry for exporting the metrics.
Thanks - got it working. I think this should be an update documentation, because outside of that blog post I haven't seen anything on exporting the metrics. Even something along the lines of "An additional http server integration is required to export the metrics" would have tipped me off. Maybe I just didn't see the right places though. Feel free to close now.
Thanks a ton for all the help.
Glad it worked. I've created an issue #677