Crowd-Funding-Bots
SET UP
pip install scrapy
pip install selenium
- Download Selenium webdriver for Chrome
USAGE
1. kickstarter.com
a) cd KickstarterDotComScraper
b) scrapy list
c) scrapy crawl kickstarterleads -o nameOfFile.json
1.1 Limitation:
- Collects info for the first 240 projects under the filter 'Nearly Funded' only
1.2 Solution (to collect info about ALL projects):
- Replace lines 6 and 10 on kickstarterleads.py with: baseURL = "https://www.kickstarter.com/discover/advanced?google_chrome_workaround&woe_id=0&sort=magic&seed=2541734&page={0:05d}
2. crowdsupply.com
a) cd CrowdSupplyDotComScraper
b) scrapy list
c) scrapy crawl crowdsupplyyields -o nameOfFile.json
2.1 Limitations:
- Collects info for the projects under the filters 'Archived', 'Crowdfunding, 'Available' only
2.2 Solution (to collect info about ALL projects:
- To 'start_urls' on line 7 of CrowdSupplyYields.py add links to other categories e.g. 'https://www.crowdsupply.com/computers-and-networking'
3. experiment.com
a) cd ExperimentDotComScraper
b) Change path to Chrome Webdriver on line 77
c) python experimentDotComScraper.py
3.1 Limitations:
- Collects info about 180 projects only
3.2 Solution (to collect info about ALL projects):
- Remove 'if-else' block checking whether k>=30 at line 35 of experimentDotComScraper.py