Network Timeout Error On Large Queries

[straley]

Hi,

I'm getting a FetchError: network timeout error. I found this goes away, if I add an exceptionally long timeout at the lines

scrape-twitter/src/lib/query.js

Line 53 in 5584121

return fetcher(resource, { agent: https.globalAgent })

and

scrape-twitter/src/lib/query.js

Line 61 in 5584121

return fetcher(resource, { agent: https.globalAgent })

, like:

    return fetcher(resource, { agent: https.globalAgent, timeout: 120000 })

It would be great to have a setting to set this timeout or make it part of the TweetStream options.

[sebinsua]

Hi, I'm going to make it so that you can pass in the environment variable SCRAPE_TWITTER_TIMEOUT (but set this to 10s by default.)