The downloader should use a multithread implement for better performance

Question

The downloader should use a multithread implement for better performance

monkey2000 opened this issue 9 years ago · 3 comments

monkey2000 commented 9 years ago

Answer 1 · 2015-07-27T18:13:21.000Z

Thanks for the suggestion! It's easy to do this, but last time I checked Wikipedia servers blocked you if you tried to open multiple download streams simultaneously. Do you have reason to believe this isn't true?

Answer 2 · 2015-08-01T04:20:22.000Z

The official repo said that max per-ip connections is 2. Also, there are many mirrors. I think fetch from multi-mirrors can solve this problem.
See also: http://dumps.wikimedia.org/

Answer 3 · 2015-08-01T04:20:35.000Z

The mirrors list can be found here: http://dumps.wikimedia.org/mirrors.html