Experiments with Redis Bloom and the Text from "Moby Dick"

This repository contains a small example using the text from the book "Moby Dick" as a data source to compare and contrast the benefits and tradeoffs of using different Redis data structures. We'll use the deterministic Set data structure, and four probabilistic data structures: Hyperloglog, Bloom Filter, Top-K and Count-Min Sketch. These probabilistic data structures generally use hashing to be more memory efficient at the cost of some accuracy and the ability to recall data added to them.

The Set and Hyperloglog are built into core Redis, to use the Bloom Filter, Top K and Count-Min Sketch data structures, you'll need Redis Stack or the RedisBloom module.

Check out our video on YouTube where Justin and Simon talk about counting things at scale with Redis Stack, using examples from this repository.

We also added some extra examples, which are currently on the adds-other-examples branch of this repository.

Prerequisites

To try out this project yourself, you'll need:

Redis Stack (installed locally, available free in the cloud, or use the supplied Docker Compose file).
Docker Desktop (if using the Docker Compose file).
git command line tools.
Node.js version 14.8 or higher (I've tested the code using Node.js version 16.4.2).
OR
Python version 3.6 or higher (I've tested the code using Python version 3.10.6).

Get the Code

To get the code, clone the repo to your machine:

$ git clone https://github.com/redis-developer/redisbloom-moby-dick.git
$ cd redisbloom-moby-dick/

Application Setup

This project has two versions, one written in NodeJS and other in Python. For the NodeJS version, head to the nodejs directory by doing:

$ cd nodejs/

and follow the instructions given in the README file inside the directory.

For the Python version, head to the python directory by doing:

$ cd python/

and follow the instructions given in the README file inside the directory.

Further Exercises

Try using the following commands to check if a given word is in the Set, or possibly in the Bloom Filter:

SISMEMBER: Find out for sure whether a word is in the Set whose key is mobydick:words:set. Try checking multiple words at once with SMISMEMER
BF.EXISTS: Find out whether a word may be or is absolutely not in the Bloom Filter whose key is mobydick:words:bloom. Try checking multiple words at once with BF.MEXISTS

Why can't we say with absolute certainty that a word is in the Bloom Filter?

Licensing

The code is made available under the terms of the MIT license. I obtained the original text file of the book "Moby Dick" from Project Gutenberg. This text is in the public domain and available in its original form here.