SOFTowaha/honeybee

🍯🐝 Distributed YouTube live chat events collector

🍯🐝 honeybee

Honeybee is a distributed YouTube live chat and moderation events collector.

How it works

• Fetch streams index from Holodex (every 10 minutes)
• Queue newly scheduled streams to a job pool
• One of the cluster members takes it and start collecting events

Spin up a cluster

cp .env.placeholder .env
vim .env

docker swarm init --advertise-addr $(curl -s https://api.ipify.org)
docker network create -d overlay --attachable honeybee
docker stack deploy -c cluster.yml hb

# in master node
make logindb

db.createUser({
  user: "worker",
  pwd: passwordPrompt(), // or cleartext password
  roles: [{ role: "readWrite", db: "honeybee" }],
});

sed -i "s/MONGO_WORKER_PASSWORD=/MONGO_WORKER_PASSWORD=<password>/" .env

Show cluster health

make health

Deploy additional cluster nodes

cd tf
terraform init
terraform apply -var total_workers=5

Teardown cluster nodes

cd tf
terraform destroy

Run one-shot task

./hb --help