Celery

A distributed task queue built for Python that processes large volumes of messages via brokers, with emphasis on real-time execution and optional scheduled tasks. Workers and clients communicate through a message broker and can retry on connection loss, while a single worker process can handle millions of tasks per minute with sub-millisecond latency when paired with RabbitMQ and optimized settings.

Celery has a large community and supports multiple brokers and result backends out of the box. It differentiates itself from alternatives like RQ or Huey through its Canvas primitives for complex workflows, native Django integration, and protocol implementations for Node.js, Go, PHP, and Rust.

Key features:

• Broker support for RabbitMQ, Redis, and Amazon SQS
• Canvas workflows with grouping, chaining, chunking, and map-reduce style patterns
• Periodic tasks via intervals or Crontab expressions
• Multiple result backends: Redis, SQLAlchemy, Django ORM, Memcached, and cloud storage options
• Rate limiting and per-task time limits
• First-class Django integration and framework hooks for Flask, Pyramid, and others

Typical scenarios: offloading email sends and report generation from web request paths, running batch processing jobs, scheduling recurring maintenance tasks, and building data pipelines. Developers use it to keep HTTP responses fast while delegating heavy work to worker pools, and benefit from Flower for monitoring and the built-in retry and chord primitives for reliability.