manestay/novel-chapter-dataset

cliffsnotes is broken

Closed this issue · 2 comments

dominik-schwabe commented

novel-chapter-dataset/scraping/cliffsnotes_scrape.py

Line 24 in 4c4e3a8

BOOKS_LIST = 'https://www.cliffsnotes.com/literature?filter=ShowAll&sort=TITLE'

The link to the books list shows only the books starting with 'A'. Therefore a lot of books are not scraped.
For me the link https://www.cliffsnotes.com/literature%3Ffilter=ShowAll&sort=TITLE (the ? is replaced with %3F) worked instead.

manestay commented

Hi dominik-schwabe, just seeing this issue now. You're right, the link was out-of-date. I've added a new commit with your fix -- really appreciate it!

manestay commented

It seems CliffsNotes updated their site, so that the correct link is now https://www.cliffsnotes.com/literature?filter=ShowAll&sort=TITLE, and the %3F doesn't work anymore. I will update this in a commit soon.