Twitch Data Exporter
Table of Contents
About The Project
Twitch data Exporter is a small app that allows to export stats from certain channels and store them on your preferred storage method and analyze them later with other tools such as Grafana.
I came up with the idea of this project while streaming and contributing to the galician language streaming community on Twitch. We wanted to show how the community was growing and some major stats for all the channels so I decided to make a small app that would allow to export this data from Twitch API to any storage (PostgreSQL at first), and later use the data on other tools as Grafana to show the relevant stats.
It currently allows the storage of:
- Channels information and channel views & followers evolution.
- Channels clips with total views for each one.
- Channels streamings, with the viewers recorded each minute(or the time range you set)
The implemented storages as of now are the following:
Miss the one you wanted? Request a feature or consider contributing to the project!
Built With
Getting Started
This is an example of how you may give instructions on setting up your project locally. To get a local copy up and running follow these simple example steps.
Prerequisites
Project is build on node and uses many npm packages, You need to set up:
- NodeJS: Follow the instructions on the webpage.
- npm
npm install npm@latest -g
Installation
- Setup your Twitch App and get your App ClientId & Client secret from Twitch Developer Console.
- (Optional) Setup a Google Sheets App if you want to retrieve channels from a spreadsheed. Follow the Node Quickstart Guide to download
credentials.json
. - Clone the repository
git clone https://github.com/pvillaverde/twitch-data-exporter
- Install NPM packages dependencies
npm install
- Copy
config_example.js
asconfig.js
and enter your twitch API clientId & clientSecret, as well as the content of credentials.json ongoogle_credentials
if you want to use spreadsheets.twitch: { clientId: 'REQUIRED', // Twitch App ClientID clientSecret: 'REQUIRED', // Twitch APP ClientSecret channels: null, // null if using google spreadsheet }, google_credentials: {/*credentials.json*/},
- Setup the storage you want to use, by default postgres, and change the connectionString to match your database connection info.
storage: 'postgres', // Which storage will be used storageConfig: { postgres: { connectionString: 'postgresql://username:password@server:port/database', } },
- Start the app by runing
npm start
ornode app/index.js
and follow the steps for google Api auth if you are using spreadsheets. - If you want to launch a docker, once you have all setup and ready, build the image and run it with:
docker build -t pvillaverde/twitch-data-exporter . docker run --name twitch-data-exporter -d pvillaverde/twitch-data-exporter
Usage
As you have seen previously, you can choose how the app learns which channels to check from 2 options:
- Defining the
twitch.channels
variable onconfig.js
with an array of channels name, ex://[clankirfed,twitch_en_galego]
- Setting up Google Sheets API and specifying a Google Spreadsheet on
config.js
which shall have the channels names on the first column:
google_spreadsheet: {
id: '1AFbvk9SLOpOyST4VWG6IOkiMdclzExUPQrKUBuEUHKY', // Google Spreadsheet ID
range: 'Canles!A2:A', // Sheet & Range
headers: 'name', // Column Headers (Optional, only first one will be used)
},
Customizing refresh intervals
Lastly, you can choose when to retrieve each data on the cron
option of config.js
, by default it is setup like this:
// Minutes Hours DayOfMonth Month DayOfWeek
fetchUsers: '0 1 * * *', // User Channels and total views will be refreshed every day at 1AM
fetchClips: '0 2 * * *', // Channel Clips will be refreshed every day at 2AM
fetchGames: '0 3 * * *', // Missing Games from the database will be fetched every day at 3AM
fetchFollows: '0 4 * * *', // Channel followers will be refreshed every dat at 4AM
fetchStreams: '* * * * *', // Every minute will who's someone on stream and viewers count.
These intervals can be customized using the cron syntax:
* * * * * *
| | | | | |
| | | | | day of week
| | | | month
| | | day of month
| | hour
| minute
second ( optional )
Roadmap
Check out the open issues page for a list of proposed features (and known issues).
Contributing
If you want to add any missing feature or storage that feed your needs, go ahead! That's what make the open source community shines, by allowing us to grow and learn from each other creating amazing tools! Any contribution you make is greatly appreciated.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature
) - Commit your Changes (
git commit -m 'Add some AmazingFeature'
) - Push to the Branch (
git push origin feature/AmazingFeature
) - Open a Pull Request
Authors
Pablo Villaverde Castro - @clankirfed
Acknowledgements
- Twitch API
- Google Sheets API
- How to use Node Cron
- TOC Generator
- Dockerizing NodeJs WebApp
- Building Efficient NodeJS DockerFiles
License
Distributed under the GNU GPL-v3 License. See LICENSE
for more information.