/wikidl

WikiDL is an efficient wikipedia data dump downloader for researchers. It uses multiprocessing for maxing out the bandwidth and CPUs, and is friendly with task schedulers like Slurm.

Primary LanguagePythonGNU General Public License v2.0GPL-2.0

Stargazers