/lambda-pond

Python3 process pool for environments without semaphore support

Primary LanguagePythonMIT LicenseMIT

lambda-pond

A Process Pool for environments where semaphore support is unavailable (sem_open and sem_init), such as AWS Lambda.

Features

  • Task Timeout
  • Worker recycling after a certain amount of processed tasks
  • TODO: Memory monitoring
  • TODO: batch / map functionality

Examples

import hashlib

from lambda_pond import Pool

def f(x):
    return hashlib.sha1(('h' * (x + 1) * 10**8).encode()).hexdigest()        

with Pool(task_timeout=2) as p:
    tasks = []
    for n in range(10):
        t = p.apply_async(f, n)
        tasks.append(t)

for task in tasks:
    print(task.identifier, '->', task.result)

References

Credits

  • API and multithreaded design using Queues inspired by the standard library multiprocessing.Pool.
  • Worker pool concepts (timeout) inspired by Celery.