Crinibus/scraper

Issues

• Need web scraping job done (for hire)

  #225 opened 7 months ago by krishmoran
  0

• Very good work! Possibility of adding other sites?

  #247 opened 8 months ago by inznn
  1

• json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

  #245 opened 9 months ago by ZhujingJava
  0

• sqlite3.IntegrityError: NOT NULL constraint failed: products.product_code

  #235 opened 10 months ago by salmansarwar5102
  2

• Can you add support for shein.com?

  #240 opened 10 months ago by georgerabus
  1

• Error with adding Elgiganten links after update and new feature request

  #232 opened a year ago by Trixarn
  1

• Possibly of telegram notifications of the products ?

  #231 opened a year ago by rokipet
  2

• Api link elgiganten.se

  #228 opened a year ago by Trixarn
  2

• Really good job but possibility for notifications?

  #227 opened a year ago by Trixarn
  2

• Add a progress bar when scraping

  #180 opened 2 years ago by Crinibus
  0

• Product info is not valid - Amazon - error getting product currency

  #207 opened 2 years ago by heya0
  3

• Move timeout for requests into settings.ini

  #202 opened 2 years ago by Crinibus
  0

• Product info is not valid - Amazon - error getting product price

  #204 opened 2 years ago by heya0
  3

• Show highest and lowest price when viewing graph

  #122 opened 4 years ago by Crinibus
  1

• Add argument to delete or reset specific (or all) products or categories

  #175 opened 2 years ago by Crinibus
  0

• Add functionality to compare two or more different products on the same graph

  #158 opened 2 years ago by Crinibus
  0

• Add argument to print all product names

  #164 opened 2 years ago by Crinibus
  0

• Add an arguments to just view the latest datapoint for a product

  #162 opened 2 years ago by Crinibus
  0

• Could newegg be added to the list of sites

  #151 opened 3 years ago by EDLLT
  2

• Anti Bot and Captcha Support

  #146 opened 3 years ago by slashdotblake
  0

• Have you considered packing this functionality in a web server?

  #145 opened 3 years ago by beruic
  1

• Split the methods "get_part_num" and "shorten_url" into the domain classes

  #99 opened 4 years ago by Crinibus
  0

• Add so a user can add multiple links at once

  #110 opened 4 years ago by Crinibus
  1

• Change so the domains saved in records.json is without "www." and ".com"

  #124 opened 4 years ago by Crinibus
  0

• Maybe add threading/multiprocessing or something similar

  #100 opened 4 years ago by Crinibus
  0

• Use plotly or something more prettier to visualize data

  #76 opened 4 years ago by Crinibus
  1

• Seperate the codes into more files for better readability

  #96 opened 4 years ago by Crinibus
  0

• Make function "check_arguments" in "add_product.py" smarter and more compact

  #94 opened 4 years ago by Crinibus
  0

• Remove all links in scraping.py and data from records.json

  #98 opened 4 years ago by Crinibus
  0

• Add a dictionary with all the domains

  #93 opened 4 years ago by Crinibus
  0

• Add ability to scrape mm-vision.dk

  #88 opened 4 years ago by Crinibus
  1

• Add ability to scrape Coolshop.dk

  #89 opened 4 years ago by Crinibus
  1

• Add ability to scrape Sharkgaming.dk

  #90 opened 4 years ago by Crinibus
  1

• Add ability to scrape Expert.dk

  #86 opened 4 years ago by Crinibus
  1

• Add ability to scrape Wupti.com

  #87 opened 4 years ago by Crinibus
  1

• Add ability to scrape Power.dk

  #82 opened 4 years ago by Crinibus
  1

• Add OBS to both README about how to change eBay link to a "valid" one

  #84 opened 4 years ago by Crinibus
  0

• Add ability to scrape eBay.com

  #64 opened 4 years ago by Crinibus
  0

• Add ability to scrape Amazon.com

  #63 opened 4 years ago by Crinibus
  0

• Add headers in requests.get calls

  #68 opened 4 years ago by Crinibus
  0

• Add ability to scrape Av-Cables.dk

  #65 opened 4 years ago by Crinibus
  0

• A way to visualize data from records.json

  #49 opened 4 years ago by Crinibus
  1

• Add ability to scrape avxperten.dk

  #46 opened 4 years ago by Crinibus
  0

• Make a optional flag to add_product.py, so only certain domains gets added to records.py for the new product

  #32 opened 4 years ago by Crinibus
  0

• Make so add_product.py can use arguments to get category and url

  #30 opened 4 years ago by Crinibus
  0

• Add ability to scrape prices from elgiganten.dk

  #22 opened 4 years ago by Crinibus
  0

• Add ability to scrape prices from power.dk

  #21 opened 4 years ago by Crinibus
  2

• Change soup.find_all(...)[0] to soup.find(...)

  #8 opened 4 years ago by Crinibus
  0