mlh758/go-redis-queue

go-redis-queue

What and why

This is a redis-based queue for usage in Go. I evaluated a lot of other options before writing this, but I really didn't like the API of most of the other options out there, and nearly all of them were missing one or more of my required features.

Features

1. Ability to add arbitrary tasks to a queue in redis
2. Automatic dedupping of tasks. Multiple pushes of the exact same payload does not create any additional work.
3. Ability to schedule tasks in the future.
4. Atomic Push and Pop from queue. Two workers cannot get the same job.
5. Sorted FIFO queue.
6. Can act like a priority queue by scheduling a job with a really old timestamp
7. Well tested
8. Small, concise codebase
9. Simple API

Usage

Adding jobs to a queue.

import "github.com/AgileBits/go-redis-queue/redisqueue"

c, err := redis.Dial("tcp", "127.0.0.1:6379")
if err != nil { ... }
defer c.Close()

q := redisqueue.New("some_queue_name", c)

wasAdded, err := q.Push("basic item")
if err != nil { ... }

queueSize, err := q.Pending()
if err != nil { ... }

wasAdded, err := q.Schedule("scheduled item", time.Now().Add(10*time.Minute))
if err != nil { ... }

A simple worker processing jobs from a queue:

c, err := redis.Dial("tcp", "127.0.0.1:6379")
if err != nil { ... }
defer c.Close()

q := redisqueue.New("some_queue_name", c)

for !timeToQuit {
  job, err = q.Pop()
  if err != nil { ... }
  if job != "" {
    // process the job.
  } else {
    time.Sleep(2*time.Second)
  }
}

A batch worker processing jobs from a queue:

c, err := redis.Dial("tcp", "127.0.0.1:6379")
if err != nil { ... }
defer c.Close()

q := redisqueue.New("some_queue_name", c)

for !timeToQuit {
  jobs, err := q.PopJobs(100) // argument is "limit"
  if err != nil { ... }
  if len(jobs) > 0 {
    for i, job := range jobs {
      // process the job.
    }
  } else {
    time.Sleep(2*time.Second)
  }
}

Requirements

• Redis 2.6.0 or greater
• github.com/garyburd/redigo/redis
• Go