/simple-task-queue

asynchronous task queues using python's multiprocessing library

Primary LanguagePythonMIT LicenseMIT

Asynchronous Task Queues in Python

Several implementations of asynchronous task queues in Python using the multiprocessing library and Redis.

Blog post: Developing an Asynchronous Task Queue in Python

Setup

  1. Fork/Clone

  2. Create and activate a virtual environment

  3. Install the dependencies

  4. Enter the Python shell and download the NLTK stopwords corpus:

    >> import nltk
>> nltk.download('stopwords')

  [nltk_data] Downloading package stopwords to
  [nltk_data]     /Users/michael.herman/nltk_data...
  [nltk_data]   Unzipping corpora/stopwords.zip.
  True

Examples

Multiprocessing Pool:

$ python simple_pool.py

Multiprocessing Queue:

$ python simple_queue.py
$ python simple_task_queue.py

Logging to a single file:

$ python simple_task_queue_logging.py

Logging to separate files:

$ python simple_task_queue_logging_separate_files.py

Redis:

$ python redis_queue.py