RedisMessageListenerContainer + IAM Auth: Keep long lived connections alive with AUTH command

[mgrundie-r7]

We are in the process of enabling IAM authentication on our ElastiCache Redis clusters.

In one of our services we are using the Lettuce client with credentials provider and RedisMessageListenerContainer to subscribe to a pub sub. When we lose connection to Redis we are forced to do some work (we override handleSubscriptionException) because we may have missed a message while the connection was down.

IAM authentication requires that the AUTH command be sent every 12 hours to prvide fresh creds and keep the connection alive. Is there any way to do this currently or can this be added. Perhaps this is should be a lettuce question rather than spring.data.redis

[mp911de]

Lettuce's RedisCredentialsProvider is the right place to obtain dynamically updated credentials. Spring Data provides a RedisCredentialsProviderFactory that you can use to implement your own way how you would construct a RedisCredentialsProvider. Lettuce calls RedisCredentialsProvider.resolveCredentials() upon connect and reconnect.

That being said, you will have to implement a scheduled service that maintains current credentials for connects and reconnects.

An IAM authenticated connection to ElastiCache for Redis will automatically be disconnected after 12 hours

To keep connections alive, you could use RedisConnectionStateListener that gets notified for each connection that is being created. With a timer, you could then send AUTH commands. However, you need to be careful as AUTH commands that would interleave with transactions may interfere with command results. If you don't use transactions, then you don't need to worry.

Keeping track of active connections can be cumbersome. The alternative would be waiting for a disconnect.

[mgrundie-r7]

Thanks for the response, I have implemented the credential provider stuff adapted form aws samples. AWS docs say MULTI EXEC transactions aren't supported with IAM authentication so I have avoided those in our other service and use Lua scripts instead.

For this specific service we are subscribing to a redis topic only and need the connection to stay alive otherwise we will have to do some expensive work each time it disconnects.

I'm looking into how to use RedisConnectionStateListener now . The beans I currently have access to are: RedisMessageListenerContainer, LettuceConnectionFactory, MessageListenerAdapter, MeterRegistry

[mgrundie-r7]

Experimenting with a main and using your RedisConnectionStateListener suggestion, I came up with the below. Any gotchas I should be aware of?

Is out of box token refreshing a feature request for spring.data.redis or for lettuce? I'am providing the credential provider when setting up the ConnectionFactory so it has everything it needs to so. Maybe users could provide an AUTH refresh parameter...

public static void main(String[] args) {

    AwsV2ClientsConfig awsV2ClientsConfig = new AwsV2ClientsConfig(Region.US_EAST_1);

    RedisProperties redisProperties = new RedisProperties();
    redisProperties.set...
    // Set redis host data, user, etc

   ...


    RedisConfiguration redisConfiguration = new RedisConfiguration(redisProperties);
    LettuceConnectionFactory lettuceConnectionFactory = redisConfiguration.redisConnectionFactory(
        redisProperties, awsV2ClientsConfig);


    var credProvider = redisConfiguration.getRedisIAMAuthCredentialsProvider(redisProperties,
        awsV2ClientsConfig);

    RedisConnectionStateListener redisConnectionStateListener = new RedisConnectionStateListener() {
      @Override
      public void onRedisConnected(RedisChannelHandler<?, ?> connection) {
        RedisConnectionStateListener.super.onRedisConnected(connection);

        new Thread(() -> {

          var lastUpdate = 0L;

          // Loop starts when connection is first opened.
          while (true) {
            if (connection.isOpen()) {

              // First iteration
              if (lastUpdate == 0L) {
                System.out.println("NEW CONNECTION");
                lastUpdate = Instant.now().getEpochSecond();
              } else {

                /*
                 * Subsequent iterations while the connection is still open
                 * todo: add check for time gap, only want to do this close to the 12hr mark
                 */
                System.out.println("REFRESH CONNECTION");

                var con = (StatefulRedisConnection) connection;

                // Get a new token from the credential provider.
                var resp = con.sync().auth(redisProperties.getRedisUser(),
                    new String(credProvider.resolveCredentials().block().getPassword()));
                lastUpdate = Instant.now().getEpochSecond();

                System.out.println(resp); // OK
              }

              try {
                // todo revise this sleep
                Thread.sleep(10000L);
              } catch (InterruptedException e) {
                throw new RuntimeException(e);
              }

            } else {

              // Exit this thread when the connection is no longer active.
              if (lastUpdate > 0L) {
                return;
              }
            }
          }
        }, "Thread-RedisConnectionRefresher").start();
      }
    };

    lettuceConnectionFactory.start();
    lettuceConnectionFactory.getNativeClient().addListener(redisConnectionStateListener);
    var connection = lettuceConnectionFactory.getConnection();

    System.out.println(connection.publish("pubsub:sometopic".getBytes(), "test".getBytes()));

    while (true) {
    }
  }

[mp911de]

I would suggest using a ThreadPoolExecutor or Lettuce's EventExecutorGroup instead of starting a new thread on each connect. onRedisConnected can be called for an existing connection as well (i.e. when reconnecting) so you should keep a mapping between RedisChannelHandler and the runnable/ScheduledFuture.

Is out of box token refreshing a feature request for spring.data.redis or for lettuce?

Not for Spring Data, but it could be one for Spring Cloud AWS.

Generally speaking, each cloud vendor does quirky things to Redis, trying to cover them all in the driver or in Spring Data would be an endless journey us trying to catch up with things that we neither have under control nor experience ourselves as we do not utilize Cloud Redis in our development or CI.

[mgrundie-r7]

Thanks, I ended up using a concurrent hash map which is populated and cleared by onRedisConnected and onRedisDisconnected then a spring @scheduled method to periodically send the AUTH command.