decrypto-org/spider

Issues

• Respect robots.txt

  #37 opened 5 years ago by jogli5er
  0

• Several Crawls in one DB

  #38 opened 5 years ago by jogli5er
  0

• Change many/many relation defined by links to contents/paths

  #26 opened 6 years ago by dionyziz
  1

• Tex compilation still contains errors

  #27 opened 6 years ago by jogli5er
  1

• Parser issue detected

  #25 opened 6 years ago by jogli5er
  0

• Remove subdomain from baseUrls and add as separate table

  #21 opened 6 years ago by jogli5er
  1

• Improve prioritizing algorithm to circumvent black holes

  #20 opened 6 years ago by jogli5er
  0

• Check why certain pages were filtered out by the blacklist

  #24 opened 6 years ago by jogli5er
  2

• Too large string exception

  #13 opened 6 years ago by jogli5er
  1

• Filter by file type in the url

  #23 opened 6 years ago by jogli5er
  0

• Classification

  #22 opened 6 years ago by jogli5er
  0

• Multiple instances

  #19 opened 6 years ago by jogli5er
  0

• Optimization: Insertion of Paths/Links

  #17 opened 6 years ago by jogli5er
  1

• Improve handling of concurrent requests to the same host

  #18 opened 6 years ago by jogli5er
  2

• Ports are stored within the base url

  #15 opened 6 years ago by jogli5er
  0

• Large numbers of paths on the same host leads to OOM condition

  #16 opened 6 years ago by jogli5er
  0

• Postgres Pool resource request times out

  #12 opened 6 years ago by jogli5er
  1

• Winston logger broke

  #14 opened 6 years ago by jogli5er
  0

• Cleanup Program

  #9 opened 6 years ago by jogli5er
  2

• Follow redirects (30x status codes)

  #10 opened 6 years ago by jogli5er
  0

• Paths & Base URLs not matching up

  #11 opened 6 years ago by jogli5er
  1

• Tests up and running

  #5 opened 6 years ago by jogli5er
  0

• Base64 Filtering

  #2 opened 6 years ago by jogli5er
  1

• Alter DB Schema

  #6 opened 6 years ago by jogli5er
  1

• Extend DB Results Cache to Pool

  #7 opened 6 years ago by jogli5er
  1

• Introduce randomizing in DB sorting

  #8 opened 6 years ago by jogli5er
  0

• Onion regex matches also non-onion URIs

  #3 opened 7 years ago by jogli5er
  2

• No relative links captured

  #4 opened 7 years ago by jogli5er
  1

• Infinite loop - detected but exception gets consumed

  #1 opened 7 years ago by jogli5er
  2