runabol/piper

Distributed computing issue when deployed on multiple nodes with kubernetes

Opened this issue · 1 comments

I have deployed piper in kubernetes using 5 piper workers/pods .

After proceeding with the tests, I have noticed that all workers are responsible for a single job which is a problem in case I want to make some POST-TRANSCODING actions with the video.

Here's an explained scenario :

  1. worker X gets the job and started transcoding the file and saving the output to a temporary file
  2. worker X finished the transcoding
  3. worker Y tries to make some POST-TRANSCODING actions with the video => Here come the problem, file not found in the worker's tmp directory.

I did a hack using a mounted NFS volume to sync the workers processing but this is very Network IO consuming.

So my question: Is it possible to bind a unique worker to a unique job ?

just curious if you have your deployment files. I am moving this direction and would like to be able to assist