codeGUST-SE/crawler

Issues

• Review and improve documentation

  #43 opened 7 years ago by evis-e
  0

• Refactor the code so that the Crawler and DocumentDatastore are two separate classes

  #42 opened 7 years ago by evis-e
  0

• Fix Issues with GitHub not getting all the code

  #41 opened 7 years ago by mustiikhalil
  1

• Use a counter instead of the :limit option of Spidr::Agent

  #40 opened 7 years ago by evis-e
  1

• Figure out the Ruby version in Gemfile

  #3 opened 7 years ago by evis-e
  0

• Use the options in Spidr::Agent

  #38 opened 7 years ago by evis-e
  1

• Determine details of geeksforgeeks input file

  #28 opened 7 years ago by evis-e
  1

• Determine details of tutorialspoint input file

  #27 opened 7 years ago by evis-e
  0

• Create PROD and DEV constants.

  #32 opened 7 years ago by evis-e
  0

• Connect the crawler to DataStore and upload crawled entries to the db.

  #7 opened 7 years ago by evis-e
  0

• Add timestamp property to store when the entry was added to datastore for debugging purposes

  #31 opened 7 years ago by evis-e
  0

• Add exception handling in the crawler

  #8 opened 7 years ago by evis-e
  1

• Get rid of non-ascii characters in the page. Write the implementation in Crawler.transform_text()

  #25 opened 7 years ago by evis-e
  0

• Determine and implement the final list of Crawlables.

  #16 opened 7 years ago by evis-e
  1

• Add score divs in Datastore.

  #15 opened 7 years ago by evis-e
  0

• Read and save <title></title> of the page in datastore.

  #22 opened 7 years ago by evis-e
  0

• Express the main_divs and score_divs with xpaths instead.

  #23 opened 7 years ago by evis-e
  0

• Replace '\n' with ' ' to cleanup page_html

  #17 opened 7 years ago by mustiikhalil
  5

• Fix the crawler taking nil values

  #14 opened 7 years ago by mustiikhalil
  1

• Build a simple crawler using spidr that prints the crawled URLs.

  #2 opened 7 years ago by evis-e
  1

• Make sure the crawler uses the politeness policy

  #9 opened 7 years ago by evis-e
  1

• Finalize the list of ignored URLs for github.

  #5 opened 7 years ago by evis-e
  1

• Finalize the list of ignored URLs for stackoverflow.

  #6 opened 7 years ago by evis-e
  1

• ignore the URL tree

  #11 opened 7 years ago by mustiikhalil
  1

• Command line interface of the crawler gets the page as an argument.

  #12 opened 7 years ago by evis-e
  0

• Build a simple crawler using spidr that simply prints the crawled URLs.

  #1 opened 7 years ago by evis-e
  1