/bloom-server

Bloom Server for Crawling@Home

Primary LanguageGo

Bloom Filter Server

Setup

  1. Install Go
  2. Install Redis & RedisBloom
  3. Clone this repository
  4. go build, to build binary
  5. ./main, to run bloom-server

Endpoints

  1. /deduplicate/, to remove duplicate from list of data. This will return list of unique data
curl -s -v -F "file=@urls.txt" -F "key=clipped" localhost:8000/deduplicate/
  1. /add/, to add urls to bloom filter from list of data. This will return 200OK if successfully added to bloom filter
curl -s -v -F "file=@urls.txt" -F "key=clipped" localhost:8000/add/
  1. /info/, to check information about duplicate count & bloom filter