AlexParra03/spiderbot

Issues

• Populate graph on chunks/streams as bot crawls

  #15 opened 6 years ago by AlexParra03
  0

• GET HTTP Request never finishes

  #13 opened 6 years ago by AlexParra03
  0

• Integrate graph/network visualization library

  #5 opened 6 years ago by AlexParra03
  2

• Send and receive JSON objects

  #12 opened 6 years ago by AlexParra03
  0

• Crawl the allow paths from robots.txt

  #11 opened 6 years ago by AlexParra03
  0

• Store endind slashes (or not) on a path

  #9 opened 6 years ago by AlexParra03
  1

• Follow 'sitemap' from robots.txt

  #10 opened 6 years ago by AlexParra03
  0

• Read from different types of URL from <a>

  #8 opened 6 years ago by AlexParra03
  1

• Handle redirections

  #7 opened 6 years ago by AlexParra03
  1

• Create iterator / generator to get all paths from internal website

  #6 opened 6 years ago by AlexParra03
  1

• Read from robots.txt

  #3 opened 6 years ago by AlexParra03
  1

• Build the directory tree of a website

  #2 opened 6 years ago by AlexParra03
  1

• Create logic to serve client application

  #4 opened 6 years ago by AlexParra03
  2