This repository contains code to replicate the no-longer-available Toronto BookCorpus dataset. To this end, it scrapes and downloads books from Smashwords, the source of the original dataset. Similarly, all books are written in English and contain at least 20k words.
- python 3.6+
- blingfire
- cachecontrol
- lxml
- requests
- tqdm
Replicating the Toronto BookCorpus dataset consists of three parts:
- Getting the download URLs of the plaintext books (optional)
- Downloading the plaintext books
- Pre-processing the plaintext books
The first part is optional, as I have already provided a list of download URLS in book_download_urls.txt ready to use. Nonetheless, you can recreate this list as follows:
python src/get_book_urls.pyProvided you have a list of download URLS in book_download_urls.txt, you can download the plaintext book as follows:
python src/download_books.pyPlease note that you have to execute the above command multiple times (~30 times to be more precise), from multiple IP-addresses, as Smashwords (temporarily) blocks any IP-address after 500 downloads. If you know of a way to automate this through python, please submit a pull request!
After downloading the plaintext books, they need to be pre-processed in order to be a true replica of the Toronto BookCorpus dataset (sentence tokenized and one sentence per line). This can be accomplished as follows:
python src/preprocess_books.pyThis project builds upon bookcorpus.
Please read the Smashwords Terms of Service carefully. Furthermore, please use the code in this repository responsibly and adhere to any copyright (and related) laws. I am not responsible for any copyright / plagiarism / legal issues that may arise from using the code in this repository.
Replicate Toronto BookCorpus is open-source and licensed under GNU GPL, Version 3.