ashishsaini/cryptopanic_scraper

cryptopanic news feed scraper

Cryptopanic Scraper

Headless chromedriver for automatic scraping of cryptopanics asynchronous newsfeed.
Explore the docs »

Report Bug · Request Feature

Table of Contents

• About the Project
  • Built With
• Getting Started
  • Prerequisites
  • Installation
• Usage
• Roadmap
• Contributing
• License
• Contact

About The Project

Cryptopanic is a crypto news aggregator that offers realtime news feeds of all things crypto as well as user input for ratings. This project was designed to scrape the data from their website so it could be later analyzed using NLP.

Built With

• Python
• Selenium

Getting Started

To get a local copy up and running follow these simple steps.

Prerequisites

• python 3
• pip

Installation

1. Clone the cryptopanic_scraper

   git clone https:://github.com/grilledchickenthighs/cryptopanic_scraper.git

2. Change directory

   cd cryptopanic_scraper

3. Install packages

   pip install -r requirements.txt

Usage

Simply run:

python cryptopanic_scraper.py --headless

If you want to see it in action, run the script without any flags.

python cryptopanic_scraper.py 

If you want to filter the type of news to scrape add the --filter flag and choose a type. {all,hot,rising,bullish,bearish,lol,commented,important,saved}

python cryptopanic_scraper.py --filter hot

You can always use the --help flag if you forget these commands:

python cryptopanic_scraper.py --help

usage: cryptopanic_webdriver.py [-h] [-v]
                                [-f {all,hot,rising,bullish,bearish,lol,commented,important,saved}]
                                [-s]

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         increase output verbosity
  -f {all,hot,rising,bullish,bearish,lol,commented,important,saved}, --filter {all,hot,rising,bullish,bearish,lol,commented,important,saved}
                        Type of News filter
  -s, --headless        Run Chrome driver headless

If your interested in analyzing the data:

Please feel free to check out the jupyter directory for getting started.

Roadmap

See the open issues for a list of proposed features (and known issues).

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Paul Mendes - @BTCTradeNation - paulsperformance@gmail.com

Project Link: https://github.com/grilledchickenthighs/cryptopanic_scraper