Can't get Redis store to work

Question

Can't get Redis store to work

Closed this issue 13 years ago · 16 comments

I've added the following to my redis.rb initializer:
uri = URI.parse(ENV['REDISTOGO_URL'] || 'redis://localhost:6379')
$redis = Redis.new(host: uri.host, port: uri.port, password: uri.password)

and the following to my girl_friday.rb initializer:

SYNC_FRIENDS_QUEUE = GirlFriday::WorkQueue.new(:sync_friends, size: 2, store: GirlFriday::Store::Redis, store_config: [{ redis: $redis }]) do |msg|
Account.sync_friends(msg[:account_id])
end

However, when I push stuff to the queue (SYNC_FRIENDS_QUEUE.push(account_id: account.id))), it gets processed in memory. Tailing the Redis log shows no connected clients. Only when I do GirlFriday.status can I see the Redis client connected. Am I configuring something wrong?

Answer 1 · 2011-09-14T02:39:08.000Z

You have to connect Redis after Unicorn forks. Check out http://j.mp/nu4zRR and see if that helps.

Are you able to get it working locally with the Redis store?

Redis.connect(url: ENV["REDISTOGO_URL"]

would work too, as connect can take a url option which will be automatically parsed.

Answer 2 · 2011-09-14T16:20:05.000Z

Thanks for the pointer. I added the following to my config/unicorn.rb file:

require 'redis'

after_fork do |server, worker|
$redis = Redis.connect(url: ENV['REDISTOGO_URL'] || 'redis://localhost:6379')
end

However, the queue still processes in memory. Only when I run GirlFriday.status from the console clients connect to Redis in the log.

Answer 3 · 2011-09-14T16:44:14.000Z

ok, I'll try to repro soon.

Answer 4 · 2011-09-16T21:59:41.000Z

Any luck reproing this yet?

Answer 5 · 2011-09-18T01:26:46.000Z

@fbjork I'm not able to reproduce. Is your app set up like this? And to clarify, are you just having problems on Heroku or local too? What command are you running? unicorn -c unicorn.rb is what I'm using & I verified that the store is Redis.

Answer 6 · 2011-09-22T20:24:37.000Z

Yep, using the new connect method worked! This is using the latest commit from master.

However, when I run GirlFriday.status I just get {} returned. Bug?

Answer 7 · 2011-09-22T23:43:27.000Z

What's good place to configure the $redis variable so I can access it from the Rails console? Since the $redis variable is only set in config/unicorn.rb it isn't set in the console.

Answer 8 · 2011-09-23T00:25:16.000Z

Wouldn't $redis be nil there anyway? Given it's set in the after_fork block inside config/unicorn.rb?

Answer 9 · 2011-09-23T00:30:01.000Z

heh, yeah, duh... I suppose you could set it in an initializer or environment file if it doesn't already exist?

$redis = Redis.connect unless $redis

?

Answer 10 · 2011-09-24T03:26:04.000Z

Ok, that worked. However on heroku I get this error:

 irb(main):009:0> GirlFriday.status
Errno::ECONNREFUSED: Connection refused - Unable to connect to Redis on 127.0.0.1:6379
    from /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:236:in `rescue in establish_connection'
    from /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:222:in `establish_connection'
    from /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:23:in `connect'
    from /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:247:in `ensure_connected'
    from /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:137:in `block in process'
    from /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:206:in `logging'
    from /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:136:in `process'
    from /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:46:in `call'
    from /app/vendor/bundle/ruby/1.9.1/gems/rpm_contrib-2.1.4/lib/rpm_contrib/instrumentation/redis.rb:25:in `block in raw_call_command_with_newrelic_trace'
    from /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.1.2/lib/new_relic/agent/method_tracer.rb:242:in `trace_execution_scoped'
    from /app/vendor/bundle/ruby/1.9.1/gems/rpm_contrib-2.1.4/lib/rpm_contrib/instrumentation/redis.rb:23:in `raw_call_command_with_newrelic_trace'
    from /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:301:in `block in llen'
    from /usr/local/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
    from /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:300:in `llen'
    from /app/vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.5/lib/girl_friday/persistence.rb:41:in `size'
    from /app/vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.5/lib/girl_friday/work_queue.rb:54:in `status'
    from /app/vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.5/lib/girl_friday.rb:43:in `block in status'
    from /app/vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.5/lib/girl_friday.rb:41:in `each'
    from /app/vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.5/lib/girl_friday.rb:41:in `inject'
    from /app/vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.5/lib/girl_friday.rb:41:in `status'
    from (irb):9
    from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.0.9/lib/rails/commands/console.rb:44:in `start'
    from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.0.9/lib/rails/commands/console.rb:8:in `start'
    from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.0.9/lib/rails/commands.rb:23:in `<top (required)>'

I've set the following in config/unicorn.rb

after_fork do |server, worker|
$redis = Redis.connect url: ENV['REDISTOGO_URL']
end

Any ideas?

Answer 11 · 2011-09-24T03:31:12.000Z

I'm actually seeing that too. A restart did not help. I actually wonder if using a global $redis might be a very bad idea? @mperham have any thoughts on this? From the little bit I've found, maybe globals aren't always global, especially when dealing with threads?

Answer 12 · 2011-09-26T02:56:02.000Z

@fbjork I just switched over to using connection_pool this evening instead of vanilla Redis instances. I'd love to get some feedback on how it works on Heroku. I'll be switching my app to using it tonight.

Answer 13 · 2011-09-26T02:59:49.000Z

Sure, how should the configuration look like given I use unicorn?

Answer 14 · 2011-09-26T05:33:05.000Z

I actually removed the after_fork block from unicorn.rb & just configured it in my app, something like this:

require 'redis'
require 'connection_pool'

$redis = ConnectionPool.new(size: 4).new{ Redis.connect(url: ENV["REDISTOGO_URL"]) }
Q = GirlFriday::WorkQueue.new(:foo, store: GirlFriday::Store::Redis, store_config: { pool: $redis } )

I tested it out on Heroku & it worked. I'm going to close this for now, but if you have more problems please open another ticket. Thanks for all your testing, we really appreciate it!

Answer 15 · 2011-09-26T21:34:09.000Z

I guess you meant to put:
$redis = ConnectionPool.new(size: 4) { Redis.connect(url: ENV["REDISTOGO_URL"]) }

When I tried to run GirlFriday.status I get this now:

LocalJumpError: no block given (yield)
from /app/vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.6/lib/girl_friday/persistence.rb:56:in block in redis' from /app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/connection_pool.rb:41:inwith'
from /app/vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.6/lib/girl_friday/persistence.rb:55:in redis' from /app/vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.6/lib/girl_friday/persistence.rb:44:insize'
from /app/vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.6/lib/girl_friday/work_queue.rb:55:in status' from /app/vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.6/lib/girl_friday.rb:43:inblock in status'
from /app/vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.6/lib/girl_friday.rb:41:in each' from /app/vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.6/lib/girl_friday.rb:41:ininject'
from /app/vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.6/lib/girl_friday.rb:41:in `status'

Answer 16 · 2011-09-26T21:47:13.000Z

Please file a new issue & I'll take a look asap.