farkaskid/WebCrawler

Issues

• Anchors having id are treated as different urls

  #22 opened 6 years ago by mahesh1996
  2

• Add reporting mechanism.

  #4 opened 6 years ago by farkaskid
  4

• Store generated url graph into some storage.

  #9 opened 6 years ago by mahesh1996
  2

• Use regex to find URLs.

  #19 opened 6 years ago by farkaskid
  3

• Bound except for the domains of the URLs specified by the user.

  #11 opened 6 years ago by mahesh1996
  0

• Improve the resolution of relative URLs.

  #16 opened 6 years ago by farkaskid
  3

• Use hashing to keep track of visited URLs instead of strings to reduce the memory usage.

  #15 opened 6 years ago by farkaskid
  3

• Implement a task queue with fixed number of workers.

  #5 opened 6 years ago by farkaskid
  2

• Duplicate URLs visited when simultaneously found by 2 or more goroutines.

  #6 opened 6 years ago by farkaskid
  1

• Handle redirections to visited URLs

  #12 opened 6 years ago by farkaskid
  2

• Ability to bound by the domain name.

  #3 opened 6 years ago by farkaskid
  3

• Add documentation before its too late.

  #8 opened 6 years ago by farkaskid
  0