Provisioning Kafka instance fails

Question

Provisioning Kafka instance fails

grdryn opened this issue 3 years ago · 7 comments

On the current tip of main (da3132b), I am experiencing an issue when I try to provision a Kafka instance using ./managed_kafka.sh --create gryan. It gets stuck in a provisioning/failed state, because the Kafka brokers can't successfully start. Here's an example log: https://gist.github.com/grdryn/cc3605b8a7f92b5145e061defcf161fb#file-kafka-0-log-L726..L727

In the mas-sso Keycloak, I see a lot of cases of the following exception, which may be related?

18:24:44,033 ERROR [org.jboss.as.controller.management-operation] (management I/O-2) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
    ("subsystem" => "infinispan"),
    ("cache-container" => "keycloak"),
    ("cache" => "userRevisions")
]): org.jboss.msc.service.ServiceNotFoundException: Service service org.wildfly.clustering.infinispan.cache.keycloak.userRevisions not found
	at org.jboss.msc@1.4.11.Final-redhat-00001//org.jboss.msc.service.ServiceContainerImpl.getRequiredService(ServiceContainerImpl.java:663)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.OperationContextImpl$OperationContextServiceRegistry.getRequiredService(OperationContextImpl.java:2293)
	at org.wildfly.clustering.service@7.3.8.GA-redhat-00001//org.wildfly.clustering.service.ServiceSupplier$1.run(ServiceSupplier.java:54)
	at org.wildfly.clustering.service@7.3.8.GA-redhat-00001//org.wildfly.clustering.service.ServiceSupplier$1.run(ServiceSupplier.java:51)
	at org.wildfly.clustering.service@7.3.8.GA-redhat-00001//org.wildfly.clustering.service.PrivilegedActionSupplier.get(PrivilegedActionSupplier.java:37)
	at org.wildfly.clustering.service@7.3.8.GA-redhat-00001//org.wildfly.clustering.service.ServiceSupplier.get(ServiceSupplier.java:67)
	at org.jboss.as.clustering.infinispan@7.3.8.GA-redhat-00001//org.jboss.as.clustering.infinispan.subsystem.CacheMetricExecutor.execute(CacheMetricExecutor.java:53)
	at org.jboss.as.clustering.infinispan@7.3.8.GA-redhat-00001//org.jboss.as.clustering.infinispan.subsystem.CacheMetricExecutor.execute(CacheMetricExecutor.java:38)
	at org.jboss.as.clustering.common@7.3.8.GA-redhat-00001//org.jboss.as.clustering.controller.MetricHandler.executeRuntimeStep(MetricHandler.java:75)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:59)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:1006)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1423)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:446)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.ModelControllerImpl.lambda$executeForResponse$0(ModelControllerImpl.java:257)
	at org.wildfly.security.elytron-private@1.10.13.Final-redhat-00001//org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
	at org.wildfly.security.elytron-private@1.10.13.Final-redhat-00001//org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:255)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.ModelControllerImpl.executeForResponse(ModelControllerImpl.java:257)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.ModelControllerImpl.executeOperation(ModelControllerImpl.java:251)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeInModelControllerCl(ModelControllerClientFactoryImpl.java:275)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.access$400(ModelControllerClientFactoryImpl.java:126)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient$1.run(ModelControllerClientFactoryImpl.java:168)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient$1.run(ModelControllerClientFactoryImpl.java:163)
	at org.wildfly.security.elytron-private@1.10.13.Final-redhat-00001//org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
	at org.wildfly.security.elytron-private@1.10.13.Final-redhat-00001//org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:255)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:198)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:175)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeOperation(ModelControllerClientFactoryImpl.java:163)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.LocalModelControllerClient.execute(LocalModelControllerClient.java:54)
	at org.jboss.as.controller@10.1.21.Final-redhat-00001//org.jboss.as.controller.LocalModelControllerClient.execute(LocalModelControllerClient.java:39)
	at org.wildfly.extension.microprofile.metrics-smallrye@7.3.8.GA-redhat-00001//org.wildfly.extension.microprofile.metrics.MetricCollector.readAttributeValue(MetricCollector.java:331)
	at org.wildfly.extension.microprofile.metrics-smallrye@7.3.8.GA-redhat-00001//org.wildfly.extension.microprofile.metrics.MetricCollector.access$400(MetricCollector.java:74)
	at org.wildfly.extension.microprofile.metrics-smallrye@7.3.8.GA-redhat-00001//org.wildfly.extension.microprofile.metrics.MetricCollector$3.getValue(MetricCollector.java:205)
	at org.wildfly.extension.microprofile.metrics-smallrye@7.3.8.GA-redhat-00001//org.wildfly.extension.microprofile.metrics.MetricCollector$3.getValue(MetricCollector.java:202)
	at io.smallrye.metrics//io.smallrye.metrics.exporters.OpenMetricsExporter.createSimpleValueLine(OpenMetricsExporter.java:492)
	at io.smallrye.metrics//io.smallrye.metrics.exporters.OpenMetricsExporter.exposeEntries(OpenMetricsExporter.java:192)
	at io.smallrye.metrics//io.smallrye.metrics.exporters.OpenMetricsExporter.getEntriesForScope(OpenMetricsExporter.java:158)
	at io.smallrye.metrics//io.smallrye.metrics.exporters.OpenMetricsExporter.exportAllScopes(OpenMetricsExporter.java:109)
	at io.smallrye.metrics//io.smallrye.metrics.MetricsRequestHandler.handleRequest(MetricsRequestHandler.java:116)
	at io.smallrye.metrics//io.smallrye.metrics.MetricsRequestHandler.handleRequest(MetricsRequestHandler.java:73)
	at org.wildfly.extension.microprofile.metrics-smallrye@7.3.8.GA-redhat-00001//org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:81)
	at org.jboss.as.domain-http-interface@10.1.21.Final-redhat-00001//org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
	at org.jboss.as.domain-http-interface@10.1.21.Final-redhat-00001//org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
	at io.undertow.core@2.0.38.SP1-redhat-00001//io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
	at io.undertow.core@2.0.38.SP1-redhat-00001//io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
	at io.undertow.core@2.0.38.SP1-redhat-00001//io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
	at io.undertow.core@2.0.38.SP1-redhat-00001//io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
	at io.undertow.core@2.0.38.SP1-redhat-00001//io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
	at org.jboss.as.domain-http-interface@10.1.21.Final-redhat-00001//org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
	at org.jboss.as.domain-http-interface@10.1.21.Final-redhat-00001//org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
	at org.jboss.as.domain-http-interface@10.1.21.Final-redhat-00001//org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:717)
	at io.undertow.core@2.0.38.SP1-redhat-00001//io.undertow.server.Connectors.executeRootHandler(Connectors.java:390)
	at io.undertow.core@2.0.38.SP1-redhat-00001//io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
	at io.undertow.core@2.0.38.SP1-redhat-00001//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
	at io.undertow.core@2.0.38.SP1-redhat-00001//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
	at org.jboss.xnio@3.7.12.Final-redhat-00001//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
	at org.jboss.xnio@3.7.12.Final-redhat-00001//org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
	at org.jboss.xnio.nio@3.7.12.Final-redhat-00001//org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
	at org.jboss.xnio.nio@3.7.12.Final-redhat-00001//org.xnio.nio.WorkerThread.run(WorkerThread.java:591)

Answer 1 · 2021-08-25T18:40:51.000Z

@maleck13 Does this stack trace from Keycloak look like it could be causing the issue I'm seeing, and/or do you have any suggestions on how to fix it?

Answer 2 · 2021-08-25T18:47:05.000Z

@akoserwal

Answer 3 · 2021-08-26T08:04:50.000Z

The above error isn't the real issue. Requires configuring the tls certificate in the broker for the mas-sso instance

Caused by: java.lang.RuntimeException: Failed to fetch public keys needed to validate JWT signatures: https://keycloak-mas-sso.apps.gryan-3az.nvee.s1.devshift.org/auth/realms/rhoas/protocol/openid-connect/certs

Answer 4 · 2021-08-26T08:39:06.000Z

@akoserwal I'll check the configuration that is being applied to the kafka instance.

However, what's the Service service org.wildfly.clustering.infinispan.cache.keycloak.userRevisions not found message about?

Answer 5 · 2021-08-26T08:46:01.000Z

You can ignore it. We are seeing the same in mas-sso staging deployment. It requires a fix from the Keycloak team, but it should be an issue for installer deployment.

Answer 6 · 2021-08-27T10:15:31.000Z

@grdryn btw I never saw the Failed to fetch public keys needed to validate JWT signatures: problem, only the service org.wildfly.clustering.infinispan.cache.keycloak.userRevisions, which we know now is a none-issue..

Smoke tests are passing for me and I see the canary connecting. Are you still seeing problems?

Answer 7 · 2022-07-22T15:36:20.000Z

Let's close this!