Async Python Functions with Celery

"Celery is a great tool for scheduled function execution in python. You can also use it for running functions in the background asynchronously from your main process. However, it does not support python asyncio. This is a big limitation, because async functions are usually much more I/O efficient, and there are many libraries that provide great async support. And parallel data processing with async.gather becomes impossible in celery without async support."

"aio-celery "does not depend on the celery codebase". Instead, it provides a new implementation of the Celery Message Protocol that enables asyncio tasks and workers. It is written completely from scratch as a thin wrapper around aio-pika (which is an asynchronous RabbitMQ python driver) and it has no other dependencies It is actively developed, and seems like a great celery alternative. But there are some downsides:"

Celery lacks native asyncio support, which limits I/O efficiency and prevents use of async.gather for parallel async processing. Two third-party options add async capabilities. aio-celery is a ground-up asyncio implementation of the Celery Message Protocol built as a thin wrapper around aio-pika, offering asynchronous tasks and workers but restricting broker and backend choices and not aiming for full Celery parity. celery-aio-pool provides a custom AsyncIOPool for existing Celery 5.3+ workers, enabling async tasks by setting an environment variable and running the worker with --pool=custom.

Read at StreamHacker