let4be/crusty

Issues

• Error in redis dockerfile

  #39 opened a year ago by ErickJ3
  3

• Attaching a database

  #38 opened 2 years ago by rtrevinnoc
  2

• Lolhtml check if we can remove elements and if this saves some cpu cycles

  #31 opened 3 years ago by let4be
  1

• Concurrency auto-tuning

  #37 opened 3 years ago by let4be
  0

• Provide example configurations for various setups

  #3 opened 3 years ago by let4be
  2

• Review how we access DNS resolved addresses

  #36 opened 3 years ago by let4be
  1

• Think about the best way to support ipv6

  #10 opened 3 years ago by let4be
  0

• Domain discovery: replace ttl cache with lru

  #30 opened 3 years ago by let4be
  0

• Consider migrating to actix

  #35 opened 3 years ago by let4be
  0

• Investigate possible deadlock/hanging of clickhouse writer

  #29 opened 3 years ago by let4be
  1

• Improve robots.txt support

  #33 opened 3 years ago by let4be
  1

• lazy.sh: allow branch option

  #32 opened 3 years ago by let4be
  0

• Glitchy buffers panel in grafana dashboard

  #28 opened 3 years ago by let4be
  0

• Review config defaults

  #27 opened 4 years ago by let4be
  0

• Implement a first approximation of PageRank for Domains

  #19 opened 4 years ago by let4be
  2

• Make crawling rules configurable via config

  #17 opened 4 years ago by let4be
  0

• Simple Job metrics

  #25 opened 4 years ago by let4be
  0

• Unify clickhouse writing operations

  #26 opened 4 years ago by let4be
  0

• Add some other important buffer len graphs to dashboard

  #24 opened 4 years ago by let4be
  0

• Add WARC support

  #1 opened 4 years ago by let4be
  0

• Improve domain heuristics at domain_filter_map

  #20 opened 4 years ago by let4be
  0

• Investigate how we track errors and what is considered an error in grafana dashboard

  #21 opened 4 years ago by let4be
  2

• Rework metrics for internal channels

  #18 opened 4 years ago by let4be
  0

• Ensure proper stopping sequence on SIGTERM

  #11 opened 4 years ago by let4be
  0

• Investigate why crawling fails to start from some websites

  #16 opened 4 years ago by let4be
  0

• curl script for super-fast start

  #15 opened 4 years ago by let4be
  0

• Queue sharding support

  #4 opened 4 years ago by let4be
  1

• Evaluate docker overlay network performance influence on high volume setups

  #13 opened 4 years ago by let4be
  1

• Consider adding bind9 to docker-compose

  #12 opened 4 years ago by let4be
  1

• Make DNS resolver configurable

  #14 opened 4 years ago by let4be
  0

• Implement faster HTML parsing

  #6 opened 4 years ago by let4be
  1

• Concurrent writing to clickhouse

  #7 opened 4 years ago by let4be
  0

• Check channel buffer sizes

  #8 opened 4 years ago by let4be
  0

• Migrate job management system to Redis

  #2 opened 4 years ago by let4be
  0