Redis To Go
Closed this issue · 6 comments
I'm considering pushing all my queued items to RedisToGo.com - The free plan provides 5mb storage (plenty for queues since I only stores id's so I doubt it'll fill up faster than that it's being processed due to the :size => 10 option.
I have a question regarding this however. I have set up 5 GirlFriday::WorkQueue instances, all using RedisToGo. This would mean I'm making 5 connections with RedisToGo am I right?
You mentioned in your wiki or readme something about ActiveRecord's pool size, that you must increase it when increasing the :size option. But that has nothing to do with the amount of connections going to RedisToGo right? I'm asking this because RedisToGo allows only 10 clients to connect, looking at your source it seems that you only establish a new connection per GirlFriday::WorkQueue instance - not per thread that's actually processing jobs ( which you can increase with the :size => amount option ).
Is this correct? What's nice is that you can have multiple free plans so you could essentially just spin up a RedisToGo free instance for each GirlFriday::WorkQueue instance if this is the case.
Thanks!
Remember :size => 10 means you have 10 worker threads per process. If you have 2-3 processes, you would have 20-30 workers. The AR connection config has nothing to do with g_f's Redis usage.
The supervisor actor is the only one who uses Redis in girl_friday so you'd have one Redis connection per work queue. Of course if your workers use Redis also, the number of connections depend on whether they share a single Redis client instance or create one per thread. The Redis client is thread-safe so you can share it among workers but you'll have to watch out for contention as your number of workers increase. Essentially multi-threaded coding means that you eventually need to move to connection pools (like with AR's database connection pool) to provide a limited number of connections (e.g. 5 connections shared among 20 workers).
Gotcha - Yeah I'm actually using Mongoid with MongoDB. Redis purely for GirlFriday - not for the actual jobs. Now that you mention it, if you spawn 3 app servers it means you're still making 3* the amount of connection to the Redis server, since the supervisor is re-created 3 times I assume.
I might as well just install Redis on my server rather than use RedisToGo since it won't have a big memory impact anyway when only used to hold queued jobs, it'll also allow me to set a higher max number of connections.
Thanks for the info!
Think it would be an ok idea to modify the Redis store to accept an instance of Redis, that way the connection could be shared? Or might that mess things up more than it would help?
Or an instance of a connection pool (e.g. https://github.com/mperham/connection_pool) so 20 threads could share 5 connections for instance.
Ah, nice. Think it should accept both or just a connection_pool?
Good question. I'm not sure I have a strong opinion right now. I think accepting a connection or a connection pool should be fine for now.