propublica/upton

pagination doesn't respect sleep time

Closed this issue · 7 comments

jeremybmerrill commented

for some reason.

It should.

esagara commented

Ah so I didn't realize I was posting in the wrong area. We are coming across this issue as well. From what I have been reading in the code, it looks like the sleep time will only work if stashing is disabled. That is the only time I can find resp_and_cache[:from_resource] being set to true as required here. At least that is my take from what I am reading in the get method. Would it make sense to write the sleep into the download_from_resource! method?

jeremybmerrill commented

Lol no worries, it's whatever.

jeremybmerrill commented

Ah figured it out

fix tk shorlty

jeremybmerrill commented

bccd019#diff-9347901060f5e801766116fbb544a20dL80

The boolean here was flipped; literally telling it not to sleep when it should have told it to sleep. Silly Jeremy.

esagara commented

Hey, I am running an array of urls through the scraper with no pagination. Sleep time is being ignored. I am working on subclassing some stuff to work around this, but I thought you should know.

jeremybmerrill commented

Hmm. Can you make a gist or something so I can test it?

Also, can you make sure you're pulling the latest master from github? I didn't push an updated gem yet.

esagara commented

I pulled the master after you updated. Will send you a gist shortly.

On Thu, Dec 19, 2013 at 11:38 AM, Jeremy B. Merrill <
notifications@github.com> wrote:

Hmm. Can you make a gist or something so I can test it?

Also, can you make sure you're pulling the latest master from github? I
didn't push an updated gem yet.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/28#issuecomment-30943796
.