"can't create Thread (11)" errors using GirlFriday::Batch
fbjork opened this issue · 9 comments
I got this error on Heroku when using GirlFriday::Batch:
can't create Thread (11)
vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.4/lib/girl_friday/actor.rb:65:in initialize' vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.4/lib/girl_friday/actor.rb:65:innew'
vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.4/lib/girl_friday/actor.rb:65:in spawn' vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.4/lib/girl_friday/work_queue.rb:133:instart'
vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.4/lib/girl_friday/work_queue.rb:21:in initialize' vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.4/lib/girl_friday/batch.rb:16:innew'
vendor/bundle/ruby/1.9.1/gems/girl_friday-0.9.4/lib/girl_friday/batch.rb:16:in `initialize'
any ideas?
That's the "running out of threads" error that I ran into on Cedar. I have no idea what causes it or how to work around it.
I asked if there was a thread limit on Heroku's Cedar stack and got this answer:
We don't have a thread limit, but there is a memory limit of about 1 GB.
Is it a Heroku issue or Ruby 1.9.2?
I assume they mean a Dyno == 1GB. Every Ruby 1.9 thread allocates a 512KB stack. Java threads have a 1MB stack but that's user-tunable. I'd advise using JRuby if you are going to use threads on Heroku so you won't need to run multiple Unicorn processes which can quickly fill that 1GB limit.
Ok. Checking Dynos tab on New Relic it says around 130mb per dyno right now. Would lowering the Unicorn workers to 2 from 3 help you think?
I would assume so but of course that lowers your web request concurrency.
On Mon, Sep 19, 2011 at 10:31 AM, Fredrik Björk
reply@reply.github.com
wrote:
Ok. Checking Dynos tab on New Relic it says around 130mb per dyno right now. Would lowering the Unicorn workers to 2 from 3 help you think?
Ok, will test that. When looking at the unicorn process in Activity Monitor on OSX I can see the Thread count increase each time I call the action that triggers the Batch operation, but it never decreases. Is that expected?
If you are dynamically creating the Batch object, as you should be, I
would expect the thread count to decrease. That's really bizarre.
Put in an issue with a simple test case if you have the time and I'll
look into it.
On Mon, Sep 19, 2011 at 12:08 PM, Fredrik Björk
reply@reply.github.com
wrote:
Ok, will test that. When looking at the unicorn process in Activity Monitor on OSX I can see the Thread count increase each time I call the action that triggers the Batch operation, but it never decreases. Is that expected?
Ok, will create a simple repro. This is definitely what's happening after a while on Heroku. After restarting it works fine until it reaches some limit.