huaweicloud/spring-cloud-huawei

客户端输出Instance changed event事件日志后,5-20s时间内还能请求到已下线的实例对应的endpoint

Opened this issue · 0 comments

场景:
部署3个服务,客户端consumer,服务端provider1、provider2,均设置无损下线waitTimeForShutDownInMillis时间为15000ms,provider2正常下线,一直压测请求客户端。

现象:
客户端consumer在waitTimeForShutDownInMillis时间后,本地实例更新打印Instance changed event事件日志,还能请求到已下线的provider2。

日志:
2024-03-05 10:42:18.637 INFO 63392 --- [-discovery-task] o.a.s.s.c.client.ServiceCenterDiscovery : Instance changed event, current: revision=2b863900874b21f5992b0787e5ea1fcb51caaa8c, instances=rest://10.37.110.162:8085|basic-provider|#; origin: revision=f16c617ab3a4052d3b385d572255f5fc7ae192f6, instances=rest://10.37.110.162:8088|basic-provider|rest://10.37.110.162:8085|basic-provider|#; appId=basic-application, serviceName=basic-provider

2024-03-05 10:42:43.772 ERROR 63392 --- [nio-8082-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: org.apache.http.conn.ConnectTimeoutException: Connect to 10.37.110.162:8088 [/10.37.110.162] failed: Connect timed out] with root cause

java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[na:na]
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[na:na]
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[na:na]
at java.base/java.net.Socket.connect(Socket.java:633) ~[na:na]
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar:4.5.13]
......
2024-03-05 10:43:18.500 INFO 63392 --- [nio-8082-exec-4] access_logger : 65e686c60a7de9e3|WebMVC receive request|0:0:0:0:0:0:0:1||0|0|/sayHello
2024-03-05 10:43:18.501 INFO 63392 --- [nio-8082-exec-4] access_logger : 65e686c60a7de9e3|RestTemplate start request||10.37.110.162:8085|0|0|/sayHello

问题定位:
根据异常项目错误日志,构建demo测试发现,服务发现使用的CachingServiceInstanceListSupplier获取实例信息,即开启了spring cloud缓存机制,反向定位发现,主要是依赖的spring-cloud-starter-loadbalancer组件中引入了evictor依赖,evictor中有ConcurrentMapWithTimedEviction的实现类,从而满足defaultLoadBalancerCacheManager初始化条件。

处理方法:
1、将spring cloud的负载缓存开关设置为false

spring:
  cloud:
    loadbalancer:
      cache:
        enabled: false

2、剔除evictor的依赖

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-loadbalancer</artifactId>
    <exclusions>
      <exclusion>
        <groupId>com.stoyanr</groupId>
        <artifactId>evictor</artifactId>
      </exclusion>
    </exclusions>
</dependency>

3、删除spring-cloud-starter-loadbalancer依赖