Introducing OverflowEdge: Your Dev's Edge! This TypeScript NodeJS web scraping script fetches the latest StackOverflow questions and exports them to CSV. Stay updated and seize the opportunity to be the first to answer. Empower your development journey with StackFlowScraper! 🚀📈 #WebScraping #StayUpdated #DevEdge
- Uses Redis as an In-memory Database for faster storage and retrieval of data.
- Stores the :
- Question title,
- Unique Question Link,
- # of views of the Question,
- # of upvotes of the Question,
- # of answers to the Question,
- Date on which the Question was asked.
- Uses Selenium Web Driver for Web Scraping.
- Fetch all data concurrently with ES6 Promises.
Package | Description |
---|---|
redis | Client for redis database |
enquirer | Library to prompt input from user |
export-to-csv | Data converter from JSON to CSV |
selenium-webdriver | Web Driver for Selenium JavaScript |
Clone the project
git clone https://github.com/RoshanJose7/webscraper-node.git
Go to the project directory
cd webscraper-node
Install dependencies
npm install
Start the server
npm run start:dev
Build server
npm run build
Start the production server
npm run start