kubernetes cluster configuration or helm chart
Closed this issue · 4 comments
Seems like kubernetes would be ideal for this as you could spin up high cpu/memory nodes on demand when there is work to do and tear them down after for the best cost/speed.
Yes, I have Kubernetes in mind. However, with the current design this is not possible or hard to do. I cannot actually say whether it's possible, since I don't have a k8s cluster to play with. Maybe I could test something with Vagrant.
In any case, my current plan is to exactly enable such a setup. This is the roadmap so far:
- Use a central Redis (cluster) for tracking the state of the cluster (which nodes does what, progress, etc). Redis supports Pub/Sub messaging, so RPC-like clustercode commands should work in addition to just holding state.
- Use MongoDB to save ffmpeg/handbrake output and which files have been converted so far.
However, this will take a long time, since I need to do some re-design (which I'm doing currently anyway)
I found this today that seems to be going down a similar path to what you've described but with redis and postgres. https://github.com/n1207n/video-transcode-queue
This is interesting, didn't know that before! Quickly glancing through the code however leads me to believe that it does not have documentation or the flexibility I have in mind.
For the v2 release, which has become a Kubernetes Operator, there's now a Helm chart:
https://github.com/ccremer/clustercode/blob/25cbabb9d55200697d9e35e8df81a0cfbd5d1571/charts/clustercode/README.md