YashTotale/goodreads-user-scraper

PyPI Package (30,000+ downloads): scrapes Goodreads user data (profile, bookshelves, books, authors, etc.)

Goodreads User Scraper

Scrape Goodreads User Data: Profile, Book Shelves, Books, Authors

     

Contents

• Usage
• Arguments
  • --user_id
  • --output_dir
  • --skip_user_info
  • --skip_shelves
  • --skip_authors
• Troubleshooting
• Development
• Publishing

Usage

Using pip:

pip install goodreads-user-scraper
goodreads-user-scraper --user_id <your id> --output_dir goodreads-data

Using pipx:

pipx run goodreads-user-scraper --user_id <your id> --output_dir goodreads-data

Arguments

--user_id

• Description: The user whose data should be scraped. Find your user id using these directions.
• Required: Yes

--output_dir

• Description: The directory where all scraped data will be output.
• Required: No
• Default: goodreads-data

--skip_user_info

• Description: Whether the script should skip scraping user information.
• Required: No
• Default: False

--skip_shelves

• Description: Whether the script should skip scraping shelves.
• Required: No
• Default: False

--skip_authors

• Description: Whether the script should skip scraping authors.
• Required: No
• Default: False

Troubleshooting

Ensure that your profile is viewable by anyone:

1. Navigate to the Goodreads Account Settings page
2. Click on the Settings tab
3. In the Privacy section, under the Who Can View My Profile question, select "anyone"

Development

1. Clone the GitHub repository

   git clone https://github.com/YashTotale/goodreads-user-scraper.git

2. Run the install script

   sh scripts/install.sh

3. Make changes

4. Run the test script

   sh scripts/test.sh

Publishing

1. Create .env

   TWINE_USERNAME=<foo>
   TWINE_PASSWORD=<bar>
   

2. Run the publish script

   sh scripts/publish.sh <patch|minor|major>