limit fetch bandwidth

Question

limit fetch bandwidth

Closed this issue 13 years ago · 7 comments

We need to limit the fetches to 1.5Mbps. Implemented the facility for this in 9d8e095

Answer 1 · 2012-02-05T18:42:17.000Z

Merged into stable and testing, and deployed to both lylina.com and lylina.radiosilence.org.
Limited lylina.com to 130 kB/s
Limited lylina.radiosilence.org to 50kB/s

Answer 2 · 2012-02-23T10:35:06.000Z

Not sure if this is actually working. Need to test whether the curl option actually works on swingline, and eventually push tasks to an asynchronous queue. That part may be filed as a new issue at some point.

Answer 3 · 2012-04-11T01:47:58.000Z

enabled Accept-Encoding in curl in 01e61aa, maybe this will help a little

Answer 4 · 2012-04-11T03:13:01.000Z

Accept-Encoding may help if the fetched content wasn't being compressed before. I think we can fix this issue by rewriting fetching so that not all the feeds are fetched in parallel at the same time. Right now the fetch is overrunning PHP's max execution time of 30 seconds.

Fetching fewer feeds at once or switching to some kind of process queue should fix both issues with fetching.

Answer 5 · 2012-04-11T04:30:36.000Z

Yeah, I absolutely agree that that is the real fix, but I'd still like to test the effect encoding has when deployed to production in terms of data downloaded and server load.

Answer 6 · 2012-04-12T23:20:33.000Z

This seems to have actually brought us below the point of running an overage in terms of bandwidth, for now.

Answer 7 · 2012-04-23T15:11:37.000Z

Commit 213e89d limits the number of feeds fetched at once and seems to be working well for now. We can probably implement a better system later on, but I think we can go ahead and close the issue.