Multiprocessing, (multi)threading ,and asynchronous programming with asyncio and Celery.
Python Event-Driven architecture using RabbitMQ and pika.
- Multithreading
- Multiprocessing
- Asyncio
Quick Recap
- Sync: Blocking operations.
- Async: Independant and Non blocking operations.
- Concurrency: Making progress together.
- Parallelism: Making progress in parallel.
Parallelism implies Concurrency. But Concurrency doesn’t always mean Parallelism.
http://masnun.rocks/2016/10/06/async-python-the-different-forms-of-concurrency/
if io_bound:
if io_very_slow:
print("Use Asyncio")
else:
print("Use Threads")
else:
print("Multi Processing")
"If your code is IO bound, both multiprocessing and multithreading in Python will work for you. Multiprocessing is a easier to just drop in than threading but has a higher memory overhead. If your code is CPU bound, multiprocessing is most likely going to be the better choice—especially if the target machine has multiple cores or CPUs. For web applications, and when you need to scale the work across multiple machines, RQ library is going to be better for you." shorturl.at/ejnz1
Speed Up Your Python Program With Concurrency : https://realpython.com/python-concurrency/
Coroutines and tasks : https://docs.python.org/fr/3/library/asyncio-task.html
Event loop : https://docs.python.org/fr/3/library/asyncio-eventloop.html
Python Asyncio: Basic Fundamentals : https://dev.to/v_it_aly/asyncio-basic-fundamentals-4i5m
Async programming in Python with asyncio : https://dev.to/welldone2094/async-programming-in-python-with-asyncio-12dl
Speed Up Your Python Program With Concurrency : https://realpython.com/python-concurrency/
Asynchronous Python for Web Development : https://stackabuse.com/asynchronous-python-for-web-development/
Async IO en Python: une solution complète : https://www.codeflow.site/fr/article/async-io-python#_un_programme_complet_demandes_asynchrones
Celery Docs : https://docs.celeryproject.org/en/stable/index.html
Celery Tutorial: A Must-Learn Technology for Python Developers : https://medium.com/swlh/python-developers-celery-is-a-must-learn-technology-heres-how-to-get-started-578f5d63fab3
Setting up a task queue using Celery and RabbitMQ : https://medium.com/@krishnadey30/setting-up-a-task-queue-using-celery-and-rabbitmq-e73f8fd15de0
Celery: an overview of the architecture and how it works : https://www.vinta.com.br/blog/2017/celery-overview-archtecture-and-how-it-works/
Celery in the wild: tips and tricks to run async tasks in the real world : https://www.vinta.com.br/blog/2018/celery-wild-tips-and-tricks-run-async-tasks-real-world/
Dealing with resource-consuming (Time + Memory) tasks on Celery : https://www.vinta.com.br/blog/2018/dealing-resource-consuming-tasks-celery/
Celery tasks Checklist : https://devchecklists.com/celery-tasks-checklist/
How to Use Celery and RabbitMQ with Django : https://simpleisbetterthancomplex.com/tutorial/2017/08/20/how-to-use-celery-with-django.html
Concurrency Type | Switching Decision | Number of Processors |
---|---|---|
Pre-emptive multitasking (threading ) |
The operating system decides when to switch tasks external to Python. | 1 |
Cooperative multitasking (asyncio ) |
The tasks decide when to give up control. | 1 |
Multiprocessing (multiprocessing ) |
The processes all run at the same time on different processors. | Many |
Concurrency Type | Features | Use Criteria | Metaphor |
---|---|---|---|
Multiprocessing | Multiple processes, high CPU utilization. | CPU-bound | We have ten kitchens, ten chefs, ten dishes to cook. |
Threading | Single process, multiple threads, pre-emptive multitasking, OS decides task switching. | Fast I/O-bound | We have one kitchen, ten chefs, ten dishes to cook. The kitchen is crowded when the ten chefs are present together. |
AsyncIO | Single process, single thread, cooperative multitasking, tasks cooperatively decide switching. | Slow I/O-bound | We have one kitchen, one chef, ten dishes to cook. |
From : https://www.vinta.com.br/blog/2017/celery-overview-archtecture-and-how-it-works/