/Crowd-Funding-Bots

Bots to access get info about creators off Kickstarter, CrowdSupply and Experiment

Primary LanguagePython

Crowd-Funding-Bots

SET UP

  1. pip install scrapy
  2. pip install selenium
  3. Download Selenium webdriver for Chrome

USAGE
1. kickstarter.com
a) cd KickstarterDotComScraper
b) scrapy list
c) scrapy crawl kickstarterleads -o nameOfFile.json

1.1 Limitation:
- Collects info for the first 240 projects under the filter 'Nearly Funded' only
1.2 Solution (to collect info about ALL projects):
- Replace lines 6 and 10 on kickstarterleads.py with: baseURL = "https://www.kickstarter.com/discover/advanced?google_chrome_workaround&woe_id=0&sort=magic&seed=2541734&page={0:05d}

2. crowdsupply.com
a) cd CrowdSupplyDotComScraper
b) scrapy list
c) scrapy crawl crowdsupplyyields -o nameOfFile.json

2.1 Limitations:
- Collects info for the projects under the filters 'Archived', 'Crowdfunding, 'Available' only
2.2 Solution (to collect info about ALL projects:
- To 'start_urls' on line 7 of CrowdSupplyYields.py add links to other categories e.g. 'https://www.crowdsupply.com/computers-and-networking'

3. experiment.com
a) cd ExperimentDotComScraper
b) Change path to Chrome Webdriver on line 77
c) python experimentDotComScraper.py

3.1 Limitations:
- Collects info about 180 projects only
3.2 Solution (to collect info about ALL projects):
- Remove 'if-else' block checking whether k>=30 at line 35 of experimentDotComScraper.py