The producer and consumer problem is an extremely popular programming paradigm. It is used for mult-process synchronization between more than one processes. There can be one producer and one consumer or multiple producers and multiple consumers. The data structure commanly used to solve this problem is a blocking queue. The producer enqueues items inside the queue and the consumer dequeues items from the queue.
Our project goal is to show the practical application of blocking data structures in long running tasks. We are creating a Python application that will have a blocking queue represented by Redis, producers which are represented by the user, and consumer which are represented by Redis workers. Redis is an in-memory data structure store. Redis supports different kinds of abstract data structures such as lists, sets, stacks, and queues. Our main interest is being able to parallelize tasks with a queue and multiple workers. The tasks that are queued will be long tasks such as web-scrapping, video editing, and reading fron a csv.
git clone https://github.com/RicardoMangandi/cop4520-producer-consumer.gitInstall FFMPEG for your operating system: https://www.ffmpeg.org/download.html
Create a Python virtual environment and activate it. Once activated use the package manager pip to install the required libraries used in the application.
pip install -r requirements.txtNext we will need to install redis on your local machine to be able to run our program and build the in-memory queue. Please follow the installtion guidelines dependent on your OS.
Once redis is installed run the command to start up the redis data structure store locally based on your local machine and requirements. You should be able to run the following command:
redis-cliThe command above runs the Redis Command Line Interface. To exit, enter exit
The Redis Dashboard is UI served on Flask that displays all the queues inside Redis and all the "worker" or "threads" listening to that queue.
In the image above we can see that there is a queue names "queue_one" and there are three working threads listening to "queue_one". This example does not do anything interesting, they just sleep for a period of time. However, there are three threads accomplishing this work in parallel and once they are done they go on and continue to the next task in the queue. The play button means they are doing work and the pause button means they have no work to do.
Run the following command to display the Redis Dashboard UI:
rq-dashboardThen open up the web-browser and go to localhost:9181
To run the consumers and generate "working threads" we need to run the following python script consumer_main.py
The application will ask the user how many threads they will like to spin up. We are limiting the user to a maximum of 10 threads.
The application will then ask the user on what queue they would like to listen to. If the user specifies a queue that does not exist nor has items inside it the workers will remain idle and not work. Please run this script first.
To run the producer and enqueue items to a specific queue we need to run the following python script producer_main.py
The application will ask the user on what queue they would like to listen to. Ensure that we are picking the queue that has active workers listening on that queue or else the queued items will remain in queue regardless of how many consuming threads we have.
To run the consumers and generate "working threads" we need to run the following python script consumer_main.py
The application will ask the user how many threads they will like to spin up. We are limiting the user to a maximum of 10 threads.
The application will then ask the user on what queue they would like to listen to. If the user specifies a queue that does not exist nor has items inside it the workers will remain idle and not work.
The task_def folder is where we will write the code logic for each enqueue that occurs.
This is line 7 on video_editing_logic inside task_def folder.
file_location = "/Users/ricardomangandi/Downloads/scene-2.mov"ensure FFMPEG is installed
run command to start redis based on OS
activate python virtual environment
pip install -r requirements.txt
rq-dashboardopen up localhost:9181
python3 consumer_main.pypython3 producer_main.py