redis/go-redis

Pipelined thread safe but not Pipeline

jfjm1 opened this issue · 1 comments

I have recently ran into an issue where different Go routines create their own Pipeline and added commands to their respective pipeline. However, when executing, the results appears to be mixed. When moved to Pipelined model, everything appear to be correct. According to the reference, it's safe for concurrent use by multiple goroutines

Expected Behavior

Based on documentation, these 2 should behavior similiarly. Pipelined is merely an alternative to Pipeline with Exec called upon function exit.

Current Behavior

Pipelined appears to be thread safe, but pipeline is not.

Steps to Reproduce

  1. Create multiple Clients, each connected to a unique Replica in a cluster.
  2. Sent ReadOnly successfully.
  3. Each client runs the following:
    1. Scan keys
    2. For batch of 10 keys, run the following:
    3. Pipelined was able to retrieve the data. But Pipeline often returns errors from node indicate MOVED

Pipelined behaviors properly:

go func(keys string) {
	results := make([]*redis.StringStringMapCmd, len(keys))
	_, err := iter.rdb.Pipelined(func(pipe redis.Pipeliner) error {
		for i, key := range keys {
			results[i] = pipe.HGetAll(key)
		}
		return nil
	})
	// processing of err and results...
}()

Pipeline does not:

go func(keys string) {
	results := make([]*redis.StringStringMapCmd, len(keys))
	pipe := iter.replica.Pipeline()
	for i, key := range keys {
		results[i] = pipe.HGetAll(key)
	}
	_, err := pipe.Exec()

	// processing of err and results...
}()

Context (Environment)

Some common use for pipeline is Set along with Expire, or optimization for multiple commands destined for the same node. Each Pipeline will only be created and executed by 1 goroutine, but there will be multiple goroutines operating on their respective pipeline.

I found the cause of the issue. It's when a Client is created, even ReadOnly is issued once immediately after the creation, it does not cover additional connections the Client may make.