/export-github-stars

A Node.js webapp to scrape the stars of specified GitHub users

Primary LanguageJavaScript

GitHub Star Export

Actions Status

Export all your GitHub stars.

preview.png

This webapp queries the GitHub API to query for the stars of a GitHub user, and stores them in an SQLite database. 10,000 stars will occupy ~6mb worth of data.

This repo uses persistent queues, and supports resumable downloading of stars. You can fetch stars of many users.

preview-jobs.png

To run

  1. Get a GitHub token from https://github.com/settings/tokens

  2. Create a .env file:

# Contents of .env

GITHUB_TOKEN=your token here

# Any name for your new database
DATABASE_NAME=output
  1. Run these commands:
npm i
make migrate-db-dev
make start

TODO:

  • Send emails using an email service about database size changes
  • Automatically update user stars every ~ few hours
  • Insert stars/repos using transactions in bulk
  • Implement ajax for pagination
  • Cache users and their star counts
  • Use http://nvd3.org/ to visualise a users stars over time
  • For many stars, should I use something like this https://github.com/tbranyen/hyperlist
  • Configure pooling correctly: knex/knex#2820 (comment) e.g.
"pool": {
  "min": 2,
  "max": 6,
  "createTimeoutMillis": 3000,
  "acquireTimeoutMillis": 30000,
  "idleTimeoutMillis": 30000,
  "reapIntervalMillis": 1000,
  "createRetryIntervalMillis": 100,
  "propagateCreateError": false // <- default is true, set to false
},