jamesturk/scrapelib

Add ability to cache via redis or memcache

Opened this issue · 2 comments

patcon commented

Any preference? My pupa scraper is running on heroku, so filesystem-based caches are non-ideal :)

Would you accept a PR for one of these options?

jamesturk commented

if you're running in production it might be best to disable as you likely
won't use the cache as it isn't intended for actual HTTP caching but
development. but if you want it, either is fine by me! much appreciated

On Aug 8, 2016 6:10 PM, "Patrick Connolly" notifications@github.com wrote:

Any preference? My pupa scraper is running on heroku, so filesystem-based
caches are non-ideal :)

Would you accept a PR for one of these options?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#34, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAfYvMc90WBO_1rqcQNBmqCVTUgiSDkks5qd6lAgaJpZM4Jfhth
.

patcon commented

Thanks! I see what you're saying, but the city of toronto started throttling the heck out of my scrapers, so I'm trying to sort out how to not restart each time I figure out the limit ;)