Fetch feeds in parallel ?

Question

Fetch feeds in parallel ?

kaihendry opened this issue 7 years ago · 24 comments

newsbeuter 2.9 - http://www.newsbeuter.org/
Copyright (C) 2006-2010 Andreas Krennmair

newsbeuter is free software and licensed under the MIT/X Consortium License.
Type `newsbeuter -vv' for more information.

Compilation date/time: Mar  8 2017 21:56:57
System: Linux 4.11.9-1-ARCH (x86_64)
Compiler: g++ 6.3.1 20170306
ncurses: ncurses 6.0.20170527 (compiled with 6.0)
libcurl: libcurl/7.54.1 OpenSSL/1.1.0f zlib/1.2.11 libpsl/0.17.0 (+libicu/59.1) libssh2/1.8.0 nghttp2/1.23.1 (compiled with 7.53.1)
SQLite: 3.19.3 (compiled with 3.17.0)
libxml2: compiled with 2.9.4

It would appear at least from the UI that refreshing feeds goes one by one. Why? It should do them all in parallel, no?

Answer 1 · 2017-07-18T03:26:40.000Z

reload-threads (parameters: ; default value: 1)
The number of parallel reload threads that shall be started when all feeds are reloaded. (example: reload-threads 3)

Answer 2 · 2017-07-18T03:51:26.000Z

Why is this a parameter? If I have 20 feeds, I expect 20 reload threads, no?

Answer 3 · 2017-07-18T04:04:08.000Z

It depends on your network's capabilities.

Answer 4 · 2017-07-18T04:05:42.000Z

I'm pretty sure my network can handle many hundreds of parallel connections like any other network...

Answer 5 · 2017-07-18T06:28:40.000Z

Did a quick search on parallel connections[1], and then checked when this was implemented[2]. The issue fixed by this commit appears to be in another bugtracker. `Firefox` now defaults to 900 max connections[3] (though this may not consider making all at once). `newsbeuter` seems to be fine checking all (110) of my feeds at once. OTOH, I had to limit the threads used by a particular software for it to work properly with my previous ISP, Time Warner. Setting `reload-threads` to `0` makes it use just 1 thread -- would be nice for it to remove the limit. Seems like a good time to raise the default, as well. [1] https://stackoverflow.com/questions/985431/max-parallel-http-connections-in-a-browser#30064610 [2] 56379ce [3] https://bugzilla.mozilla.org/show_bug.cgi?id=1260218

…

-------- Original Message -------- Subject: Re: [akrennmair/newsbeuter] Fetch feeds in parallel ? (#581) Local Time: July 18, 2017 12:05 AM UTC Time: July 18, 2017 4:05 AM From: ***@***.*** To: akrennmair/newsbeuter ***@***.***> Daniel M. Capella ***@***.***>, Comment ***@***.***> I'm pretty sure my network can handle many hundreds of parallel connections like any other network... — You are receiving this because you commented. Reply to this email directly, [view it on GitHub](#581 (comment)), or [mute the thread](https://github.com/notifications/unsubscribe-auth/ADXpfrsT7yekexNAwfdRbWr0tVKvdcIGks5sPC8XgaJpZM4Oav70).

Answer 6 · 2017-07-18T18:01:31.000Z

Really like the reload-threads 0 idea, though the default should still be kept low (perhaps 20?) because each thread takes up memory and not all of us got gigabytes of it laying around.

And we should limit the number of connections per host like browsers do, though in our case we can set it to 1. I wager that:

it's rare to fetch multiple feeds from one host; and
in such cases, the host itself is well-hosted (Google's Feedburner, BBC etc.), so it'll respond quick enough anyway.

Anyone willing to draw up a PR?

The issue fixed by this commit appears to be in another bugtracker.

Yes, it's on Google Code. No details there, though.

Answer 7 · 2017-07-18T19:10:47.000Z

should limit the number of connections per host like browsers do, though in our case we can set it to 1

An option like reload-threads-per-server would be nice for this. My feeds include ~20 YouTube feeds and ~30 GitHub feeds. Firefox still defaults to 6 here, but newsbeuter seems happy fetching these all simultaneously.

Edit: I do see a quick CPU spike across my 4 cores when reloading all feeds.

Anyone willing to draw up a PR?

If I could. :)

Answer 8 · 2017-07-18T19:33:07.000Z

An option like reload-threads-per-server would be nice for this.

I think users will just jack it up real high in hopes of getting their feeds a second earlier. Which might even not happen, i.e. it'll be slower, because of TCP slow start.

newsbeuter seems happy fetching these all simultaneously

I consider this a bug :)

Answer 9 · 2017-07-18T19:36:01.000Z

FWIW, it appears to check them all in under a second.

Answer 10 · 2017-08-04T10:26:52.000Z

Actually I've found it misses several feeds when fetching all at once.

Answer 11 · 2017-08-04T11:23:37.000Z

What do you mean by "misses"? How do you check that?

Answer 12 · 2017-08-05T00:46:39.000Z

I had enabled delete-read-articles-on-quit, which I didn't know would cause most (if not all) of my feeds to re-add articles, and some were missing until I lowered reload-threads to 30. Have not played with reload-threads since then.

Answer 13 · 2017-08-05T04:02:08.000Z

After launching just now (with auto-reload enabled), I've found at least three more feeds that "weren't previously fetched".

Answer 14 · 2017-08-09T18:03:38.000Z

I had enabled delete-read-articles-on-quit, which I didn't know would cause most (if not all) of my feeds to re-add articles

I just pushed a fix to master, can you please test? Commit ID is cc5d03e

I'm not sure what's up with articles missing if you have reload-threads set to some high(-ish) value. If you enable error-log, would you see any errors there? (I assume the problem is intermittent and running with -dnewsbeuter.log -l6 isn't an option due to the size of the log.)

Answer 15 · 2017-08-10T01:29:19.000Z

@Minoru, thank you, it does work.

Will keep an eye on the error log.

Answer 16 · 2017-08-10T19:41:31.000Z

[2017-08-10 15:28:22] Error while retrieving https://www.youtube.com/feeds/videos.xml?channel_id=UCR-QYzXrZF8yFarK8wZbHog: HTTP response code said error
[2017-08-10 15:28:22] Error while retrieving https://protonmail.com/blog/feed/: SSL connect error                      
[2017-08-10 15:28:22] Error while retrieving https://www.youtube.com/feeds/videos.xml?channel_id=UC2DjFE7Xf11URZqWBigcVOQ: HTTP response code said error
[2017-08-10 15:28:23] Error while retrieving https://www.youtube.com/feeds/videos.xml?channel_id=UC7pp40MU_6rLK5pvJYG3d0Q: HTTP response code said error

Edit: Got basically the same results at the next auto-reload. ProtonMail apparently borked SSL for their feed. No errors when I :set reload-threads 30 and manually reload-all.

Edit2: 8 YT feed errors after setting reload-threads to 30 in the config and restarting newsbeuter. None after lowering it to 20 and a subsequent restart. Only 1 error after setting it back to 30 and restarting. Not getting errors for ProtonMail anymore..

Edit3: Another auto-reload: 2 more YT errors and the ProtonMail error is back.

Edit4: I receive YT errors nearly every auto-reload. reload-threads has been set to 10 for the last two auto-reloads.

Answer 17 · 2017-08-11T19:03:30.000Z

I'll try to add the actual HTTP code to the log tomorrow—right now it's quite useless. Knowing what kind of errors it is will help understand what can/should we do about them. I wager it's 429 because you're fetching a lot of stuff simultaneously.

Answer 18 · 2017-08-12T02:51:47.000Z

Today with reload-threads set to 30:

[2017-08-11 18:33:36] Error while retrieving https://lwn.net/headlines/newrss: Peer certificate cannot be authenticated with given CA certificates
[2017-08-11 19:33:36] Error while retrieving https://lwn.net/headlines/newrss: Peer certificate cannot be authenticated with given CA certificates
[2017-08-11 20:33:37] Error while retrieving https://lwn.net/headlines/newrss: Peer certificate cannot be authenticated with given CA certificates
[2017-08-11 21:33:36] Error while retrieving https://lwn.net/headlines/newrss: Peer certificate cannot be authenticated with given CA certificates
[2017-08-11 22:33:36] Error while retrieving https://lwn.net/headlines/newrss: Peer certificate cannot be authenticated with given CA certificates

Answer 19 · 2017-08-12T10:49:16.000Z

(Edited your comment to move the log excerpt from external Pastebin into the comment itself. Had enough Paste- and imagebin links expire on me to not trust any. Nothing short of GitHub's death should take this issue tracker down :)

SSL errors are easily explained if you look at LWN's certificate—it has been issued yesterday, on August 11th. Apparently they forgot to do it and the HTTPS was down for a few hours. No worries.

Answer 20 · 2017-08-12T11:33:09.000Z

@polyzen, I just added the HTTP code to the error message. Let's see what causes those fetch failures for you!

Answer 21 · 2017-08-13T03:37:02.000Z

The error log was empty all day, until 3 404's from YT about an hour ago.

Answer 22 · 2017-08-13T09:46:52.000Z

Hmm. A fluke on YouTube's site? That's weird, but I can't do anything about it.

I'm not even sure what we're looking for anymore. Apparently reload-threads doesn't produce a ton of errors, so... problem solved?

Answer 23 · 2017-08-13T09:50:20.000Z

Thank you. Will continue to play around with the setting.

Answer 24 · 2017-08-14T03:21:13.000Z

Not seeing any noteworthy errors while fetching all feeds at once.