envoyproxy/envoy

Upgrading envoy causes JedisException: Could not get a resource from the pool

Opened this issue · 1 comments

Hi,

We have a envoy configured for a Redis cluster. The current version of it 1.18.3, and we have observed no issues. Recently, we upgraded it to 1.27.2, and we noticed errors in the clients trying to access the cache. The error is:

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
Caused by: java.util.NoSuchElementException: Unable to validate object
 at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:502) ~[commons-pool2-2.2.jar:2.2]
 at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~[commons-pool2-2.2.jar:2.2]
 at redis.clients.util.Pool.getResource(Pool.java:48) ~[ContXJedis-2.8.0.jar:?]

However, I tried adding the below config on top of the existing envoy configuration and the issue got sorted out:

upstream_connection_options:
        tcp_keepalive:
          keepalive_interval: 5
          keepalive_probes: 1
          keepalive_time: 30 

Can anyone explain the reason for this behaviour?

tyxia commented

@Vasanthan-R from your description , it seems to point to regression.
Would you either (1) do some regression check to see the difference in Envoy between 1.18.3 and 1.27.2. with the clue you found (2) Update to latest version to see if this still occurs.

I can later add the code/domain owner if you don't have luck with either of them above.