eureka server latency high(large than 30 seconds) due to all http thread blocked in EurekaInstanceMonitor(when set eureka.server.metrics.enabled=true)
Closed this issue · 14 comments
huntkalio commented
version: spring-cloud-netflix-eureka-server-4.1.3
all of eureka tomcat http thread blocked in EurekaInstanceMonitor
"http-nio-0.0.0.0-8761-exec-123" #295 [29837] daemon prio=5 os_prio=0 cpu=373.45ms elapsed=50.25s tid=0x0000ffff1809b470 nid=29837 waiting for monitor entry [0x0000ffff066ab000]
java.lang.Thread.State: BLOCKED (on object monitor)
at io.micrometer.core.instrument.MeterRegistry.remove(MeterRegistry.java:756)
- waiting to lock <0x00000000cee8c920> (a java.lang.Object)
at io.micrometer.core.instrument.MeterRegistry.remove(MeterRegistry.java:723)
at io.micrometer.core.instrument.MeterRegistry.removeByPreFilterId(MeterRegistry.java:740)
at io.micrometer.core.instrument.MultiGauge.lambda$register$0(MultiGauge.java:72)
at io.micrometer.core.instrument.MultiGauge$$Lambda/0x000000e001921e78.apply(Unknown Source)
at java.util.stream.ReferencePipeline$3$1.accept(java.base@21.0.3/ReferencePipeline.java:197)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(java.base@21.0.3/ArrayList.java:1708)
at java.util.stream.AbstractPipeline.copyInto(java.base@21.0.3/AbstractPipeline.java:509)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@21.0.3/AbstractPipeline.java:499)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(java.base@21.0.3/ReduceOps.java:921)
at java.util.stream.AbstractPipeline.evaluate(java.base@21.0.3/AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(java.base@21.0.3/ReferencePipeline.java:682)
at io.micrometer.core.instrument.MultiGauge.lambda$register$1(MultiGauge.java:82)
at io.micrometer.core.instrument.MultiGauge$$Lambda/0x000000e0019213e8.apply(Unknown Source)
at java.util.concurrent.atomic.AtomicReference.getAndUpdate(java.base@21.0.3/AtomicReference.java:188)
at io.micrometer.core.instrument.MultiGauge.register(MultiGauge.java:63)
at org.springframework.cloud.netflix.eureka.server.metrics.EurekaInstanceMonitor.onApplicationEvent(EurekaInstanceMonitor.java:72)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:452)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:385)
at org.springframework.cloud.netflix.eureka.server.InstanceRegistry.publishEvent(InstanceRegistry.java:148)
at org.springframework.cloud.netflix.eureka.server.InstanceRegistry.handleRenewal(InstanceRegistry.java:136)
at org.springframework.cloud.netflix.eureka.server.InstanceRegistry.renew(InstanceRegistry.java:105)
at com.netflix.eureka.resources.InstanceResource.renewLease(InstanceResource.java:112)
at java.lang.invoke.LambdaForm$DMH/0x000000e0014bc000.invokeVirtual(java.base@21.0.3/LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/0x000000e001a3a800.invoke(java.base@21.0.3/LambdaForm$MH)
at java.lang.invoke.Invokers$Holder.invokeExact_MT(java.base@21.0.3/Invokers$Holder)
at jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(java.base@21.0.3/DirectMethodHandleAccessor.java:157)
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(java.base@21.0.3/DirectMethodHandleAccessor.java:103)
at java.lang.reflect.Method.invoke(java.base@21.0.3/Method.java:580)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda/0x000000e00170d4a8.invoke(Unknown Source)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:696)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.ServletContainer.serviceImpl(ServletContainer.java:378)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:554)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:494)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:431)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at dev.akkinoc.spring.boot.logback.access.security.LogbackAccessSecurityServletFilter.doFilter(LogbackAccessSecurityServletFilter.kt:17)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.springframework.cloud.netflix.eureka.server.EurekaServerAutoConfiguration$1.doFilterInternal(EurekaServerAutoConfiguration.java:337)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108)
at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231)
at org.springframework.security.web.FilterChainProxy$$Lambda/0x000000e00196aac0.doFilter(Unknown Source)
at org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:479)
at org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation$$Lambda/0x000000e001980268.doFilter(Unknown Source)
at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:340)
at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation$$Lambda/0x000000e001980488.doFilter(Unknown Source)
at org.springframework.security.web.ObservationFilterChainDecorator.lambda$wrapSecured$0(ObservationFilterChainDecorator.java:82)
at org.springframework.security.web.ObservationFilterChainDecorator$$Lambda/0x000000e00196af00.doFilter(Unknown Source)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:128)
at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:100)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:213)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82)
at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter$$Lambda/0x000000e00196ed80.doFilter(Unknown Source)
at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$0(ObservationFilterChainDecorator.java:323)
at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation$$Lambda/0x000000e00196efc0.doFilter(Unknown Source)
at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:224)
at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)
at org.springframework.web.servlet.handler.HandlerMappingIntrospector.lambda$createCacheFilter$3(HandlerMappingIntrospector.java:195)
at org.springframework.web.servlet.handler.HandlerMappingIntrospector$$Lambda/0x000000e00178d5f0.doFilter(Unknown Source)
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)
at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74)
at org.springframework.security.config.annotation.web.configuration.WebMvcSecurityConfiguration$CompositeFilterChainProxy.doFilter(WebMvcSecurityConfiguration.java:230)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:113)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at dev.akkinoc.spring.boot.logback.access.tomcat.LogbackAccessTomcatValve.invoke(LogbackAccessTomcatValve.kt:55)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:731)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:389)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:904)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
at java.lang.Thread.runWith(java.base@21.0.3/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.3/Thread.java:1583)
huntkalio commented
with only several hundred registry.When latency high, other eureka node will retry to replica registry to problem eureka server.This can cause an avalanche of eureka server。
OlgaMaciaszek commented
Hello @huntkalio, thanks for reporting the issue. Have marked this as a bug and am working on a fix.
huntkalio commented
@OlgaMaciaszek Does use task executor can avoid block？I stop all other eurekaserver (only remain one problem eureka server) and wait aboud 1 hours, but it‘s thread still blocked.(all http nio thread + Eureka-EvictionTimer thread )
most thread block at:
https://github.com/micrometer-metrics/micrometer/blob/083f12e2a09e081567807f1be6273acf4da39dc5/micrometer-core/src/main/java/io/micrometer/core/instrument/MeterRegistry.java#L756
a little thread block at:
https://github.com/micrometer-metrics/micrometer/blob/083f12e2a09e081567807f1be6273acf4da39dc5/micrometer-core/src/main/java/io/micrometer/core/instrument/MeterRegistry.java#L647
here is thre thread dump:
eureka-threaddump.zip
Does it have a dead lock? If not,why it process so slow(or may we use some Non-thread-safe method?)
I just mock 700 eureka instance to register in eurekaserver, then it will reproduce。(May occur when syncing hundred registry at once) When blocked，avg several second can process only one Registry event。
OlgaMaciaszek commented
@huntkalio The change should cause an AsyncTaskExecutor to be injected (unless you're using virtual threads; are you?) and it should trigger the execution asynchronously in some other thread (javadoc). Have you tried again after the change? If yes, could you please provide the thread dump in some kind of text format, not images?
huntkalio commented
The text thread dump can see eureka-threaddump.zip ，format html
@huntkalio The change should cause an
AsyncTaskExecutorto be injected (unless you're using virtual threads; are you?) and it should trigger the executionasynchronously in some other thread(javadoc). Have you tried again after the change? If yes, could you please provide the thread dump in some kind of text format, not images?
huntkalio commented
@OlgaMaciaszek What I mean is that if it is still very slow even if it is executed in other threads, the slow execution will eventually cause the main thread to block or memory overflow (unless you discard it).
huntkalio commented
May due to when many thread register in MeterRegistry.java ，it will have pool performance。If so，do we need to update monitoring in eureka registration or heartbeat events every time? Can we update EurekaInstanceMonitor at fixed intervals to reduce lock contention?
OlgaMaciaszek commented
@huntkalio Have pushed another change.
huntkalio commented
@huntkalio Have pushed another change.
- problem is not in function collectAggregatedCounts()，but in registerMetrics().
- all thread block in it registerMetrics.
- AsyncTaskExecutor will also be blocked
May due to when many thread register in MeterRegistry.java ，it will have pool performance。If so，do we need to update monitoring in eureka registration or heartbeat events every time? Can we update EurekaInstanceMonitor at fixed intervals to reduce lock contention?
OlgaMaciaszek commented
Possibly, we might consider switching to this kind of behaviour. Could you please try it out against the current snapshots first and provide a fresh thread dump if the issue persists?
huntkalio commented
Possibly, we might consider switching to this kind of behaviour. Could you please try it out against the current snapshots first and provide a fresh thread dump if the issue persists?
@OlgaMaciaszek here is the dump for 4.1.4-SNAPSHOT
- all task thread block because contention lock （Because there are only 8 threads, competition is better than tomcat http thread）；
- because task thread consumption may cannot keep up with production，AsyncTaskExecutor block queue may continue grow until OMM ；and taskEurekaInstanceMonitor may also have delay；and this also can waste of cpu time。
- the http thread now is not blocked
OlgaMaciaszek commented
Thanks for the threa dump @huntkalio. Was it done on the latest code version? (It's now available in 4.1.4 release in Maven). Also, it seems you're using virtual threads, is that right?
huntkalio commented
Thanks for the threa dump @huntkalio. Was it done on the latest code version? (It's now available in
4.1.4release in Maven). Also, it seems you're using virtual threads, is that right?
@OlgaMaciaszek it was done on the latest code 。Not use virtual threads.
OlgaMaciaszek commented
Thanks @huntkalio, could you please also share the sample code of the app that you instantiate as a link to a separate repo with an executable app?