Autothrottling

[lekhakpadmanabh]

Instead of hitting the server every [x] seconds, crawl intervals should be determined by the frequency of incoming items (posts/comments). Based on the items crawled per crawl interval, crawl_delay should be readjusted - increased for low activity periods, decreased for high activity periods. There are several ways to implement this, ideas and volunteers are welcome.