The bot has three goals:
- Notify Open Library of new books that they may not have imported yet
- Tag NYT best sellers and reviews
- Add links to NYT reviews
The changes made by this bot are visible here. The profile mentions another repo but this is the actual code running.
This bot runs via GitHub Actions here every Wednesday at 10AM EST. You can see the results of the actions here.
PS: This workflow is automatically disabled every 60 days by GitHub unless there is activity in the repo. Thankfully, they send me an email with one button to click to re-enable it.
- Install requirements
pip install -r requirements.txt
- Set the environment variable
NYT_API_KEY
to be able to call the NYT API - Configure Open Library CLI
- Alternatively, set
OL_ACCESS_KEY
andOL_SECRET_KEY
env vars
- Alternatively, set
- Run
python nyt_bestseller_collector.py
- this will get the bestsellers for this month and put them inresult.json
- Run
python add_nyt_bestseller_tag.py --dry-run=False
to add the bestseller tags - Run
python add_nyt_review_link.py --dry-run=False
to add reviews
.env
file is supported and probably the easiest way to set env vars
NYT bot was run in August 2021 to do bulk imports for the following historical data:
- nyt_bestsellers_2008_06_03_to_2021_08_03 - scraped from public api via script here.
- all_nyt_book_review_2021_08_03 - using a NYT database dump since public apis don't make it easy to get old data.
- Best Sellers From NYT API: 20,946
- Best Sellers missing from Open Library: 3,955
- Missing bestsellers successfully imported: 370
- Subject tags added to all books found in Open Library
- Reviews From NYT API: 24,975
- Reviews missing from Open Library: 2,889
- Missing reviewed books successfully imported: 1,435
- Subject tag and link added to all books found in open library
Checklist for bestsellers:
- If no nyt subjects exist, add them ✅
- If both nyt subjects exist, make no change ✅
- If one nyt subject is missing, add the other - not yet
Checklist for reviews:
- If review doesn't exist, add it ✅
- If review subject doesn't exist, add it ✅
There is no way to get all reviews. But we can get reviews of bestsellers. As such, reviews of books aren't on the bestseller list are not added by this bot.