Memory leak with postgis adapater & Sidekiq
Closed this issue · 2 comments
Hello, I'm investigating on a memory leak. You can found a rails app here to reproduce.
I'm running this job to browse a large table with the Rails method find_each.
In this example, I try to call GC.start every 10 batches to reduce memory impact and then inspect the GC stats and current memory usage.
When the job starts:
GC.start After Before Last
:heap_available_slots 324,855 324,855 309,774
:heap_live_slots 246,979 323,461 164,291
:heap_free_slots 77,876 1,394 145,483
Memory: 75.9 MB
After dozens of occurences:
GC.start After Before Last
:heap_available_slots 12,670,227 12,670,227 12,049,043
:heap_live_slots 12,621,392 12,669,837 12,000,422
:heap_free_slots 48,835 390 48,621
Memory: 815 MB
...
GC.start After Before Last
:heap_available_slots 13,292,225 13,292,225 12,670,227
:heap_live_slots 13,242,362 13,291,861 12,621,392
:heap_free_slots 49,863 364 48,835
Memory: 851 MB
The heap size and the memory usage increase until the final blow up.
The leak happens when using the postgis adapter through Sidekiq:
- performing the job through the rails console with the gem 'activerecord-postgis-adapter' in Gemfile doesn't cause a memory leak
- performing the job through the sidekiq without the gem 'activerecord-postgis-adapter' in Gemfile
doesn't cause a memory leak - performing the job through the sidekiq with the gem 'activerecord-postgis-adapter' in Gemfile cause a memory leak
[EDIT: Memory usage reported was in MB, not in KB]
The issue comes from the loading of ActiveRecord::ConnectionAdapters::PostGIS::PostGISDatabaseTasks which inherits from ActiveRecord::Tasks::PostgreSQLDatabaseTasks
I guess that PostgreSQLDatabaseTasks is not multithread-prof, that why ActiveRecord use autoloading (provided by ActiveSupport) to require it only on needed : when calling rails tasks.
PR is coming.