crawler-commons/url-frontier

Issues

• Update API definition to follow protobuf conventions and best practices

  #110 opened 2 months ago by jdpedrie
  4

• Fix generation and pushing of image to Dockerhub

  #108 opened 3 months ago by jnioche
  1

• Ability to set Limit on domain

  #90 opened 5 months ago by zaibacu
  3

• PutURLs calls failing with java.lang.IllegalStateException

  #77 opened 2 years ago by michaeldinzinger
  2

• ignite.purge answers with 'couldn't delete workdir'

  #74 opened 2 years ago by michaeldinzinger
  0

• IgniteHeartbeat Thread throws a NullPointerException during start of Ignite service

  #72 opened 2 years ago by michaeldinzinger
  3

• Configs read.thread.num and write.thread.num are ignored for IgniteService

  #73 opened 2 years ago by michaeldinzinger
  1

• Remove Ignite implementation

  #96 opened 3 months ago by jnioche
  0

• getURLs will lock whole Queue when not assign Key

  #87 opened 3 months ago by saselovejulie
  2

• add getInfo endpoint

  #57 opened 4 months ago by jnioche
  1

• Generate multi arch Docker images

  #86 opened a year ago by jnioche
  0

• DistributedFrontierService to use threadpool for all the writes

  #69 opened 2 years ago by jnioche
  2

• Some calls to GetURLs failing with java.lang.IllegalStateException

  #71 opened 2 years ago by anjackson
  18

• Exception when trying to delete a non-existing crawl

  #70 opened 2 years ago by jnioche
  0

• Dependency updates

  #68 opened 2 years ago by jnioche
  0

• ShardedRocksDBService does not return ack when identical URLs are sent in short succession

  #67 opened 2 years ago by jnioche
  1

• java.lang.NoClassDefFoundError: com/google/common/cache/RemovalNotification when stopping ShardedRocksDBService

  #58 opened 2 years ago by jnioche
  1

• java.lang.NoClassDefFoundError: io/grpc/netty/shaded/io/netty/util/concurrent/DefaultPromise$1

  #59 opened 2 years ago by jnioche
  1

• Multithread reading from queues

  #41 opened 2 years ago by jnioche
  0

• Update version of gRPC to 1.50.0

  #61 opened 2 years ago by jnioche
  0

• Use multiple threads for putting URLs

  #63 opened 2 years ago by jnioche
  10

• Batch write operations

  #64 opened 2 years ago by jnioche
  0

• URLFrontier AckMessage does not contain IDs of ACKed URLInfo

  #62 opened 2 years ago by FelixEngl
  3

• Getting a stack dump when closing RocksDB at the end of the service

  #17 opened 3 years ago by jnioche
  3

• RocksDB backend - faster restarts

  #54 opened 3 years ago by jnioche
  1

• Exception caught when deleting queue

  #55 opened 3 years ago by jnioche
  0

• Faster recovery for RocksDB service implementation

  #52 opened 3 years ago by jnioche
  0

• Use Java 11

  #51 opened 3 years ago by jnioche
  0

• Publish service jar on Maven

  #42 opened 3 years ago by jnioche
  0

• Tuning Rockdb configuration

  #38 opened 3 years ago by jnioche
  0

• zero-padding the timestamp used in the keys for RocksDB

  #37 opened 3 years ago by jnioche
  1

• ConcurrentModificationException when calling GetStats

  #36 opened 3 years ago by jnioche
  1

• Reduce thread contention when the DB gets large

  #35 opened 3 years ago by jnioche
  0

• Odd queue names

  #34 opened 3 years ago by anjackson
  1

• GRPC error when attempting to use from Python/Scrapy

  #33 opened 3 years ago by anjackson
  1

• Stopping the Docker image with Docker kill leaves RocksDB in a corrupt state

  #32 opened 3 years ago by jnioche
  2

• Pagination of ListQueues

  #26 opened 3 years ago by jnioche
  0

• Set an explicit limit to the size of a key to 255 chars

  #27 opened 3 years ago by jnioche
  0

• Remove lock in putURLs for RocksDB service

  #28 opened 3 years ago by jnioche
  0

• exception thrown when no rocksdb.path set

  #25 opened 4 years ago by jnioche
  0

• Add GetActive endpoint to API

  #24 opened 4 years ago by jnioche
  0

• Docker container running as root

  #23 opened 4 years ago by jnioche
  0

• Client needs to throttle sending seeds to Frontier

  #22 opened 4 years ago by jnioche
  0

• Add contributions policy

  #21 opened 4 years ago by jnioche
  3

• Stats for memory implementation report the wrong number of URLs in process

  #15 opened 4 years ago by jnioche
  0

• Scan of table failing for queues which have been completed

  #16 opened 4 years ago by jnioche
  0

• Use Picocli to configure the server

  #14 opened 4 years ago by jnioche
  0

• use CLI library for parsing command line argument for Java Client

  #6 opened 4 years ago by jnioche
  4

• dummy server to support never refetch

  #8 opened 4 years ago by jnioche
  0