Celery & Task Queues
Run background tasks and scheduled jobs with Celery - task retry logic, rate limiting, chaining, beat scheduler, and monitoring with Flower.
skill: Celery & Task Queues category: Backend tags: [Celery, Python, Task Queue, Redis, Async] readingTime: 6
Move any operation taking > 200ms out of the HTTP request lifecycle - email, image processing, API calls, report generation. Celery with Redis makes this a 20-line change. Never let slow tasks degrade user experience.
Workflow
- Set up Celery with RedisRedis as both broker (queues tasks) and result backend (stores results) is the simplest production setup.
- Write tasks with retry logicNetwork calls and external APIs fail. Retry with exponential backoff prevents task loss without hammering failing services.
- Route tasks to dedicated queuesSlow tasks (video encoding) should not block fast tasks (email sending). Separate queues with dedicated workers prevent queue starvation.
- Chain and group tasksCanvas primitives let you compose complex workflows - sequences, parallel groups, and chords (group + callback).
- Schedule periodic tasks with Celery BeatBeat is a cron-like scheduler that sends tasks to Celery workers at configured intervals - no separate cron job needed.
Output Format
Celery setup: Redis broker/backend, retry with backoff, dedicated queues per task type, and beat scheduler.
tasks.py: @shared_task(bind=True, autoretry_for=(Exception,), retry_backoff=True)
queues: emails (fast), heavy (slow), default
beat_schedule: periodic tasks
Flower: celery -A myapp flower (monitoring UI)Examples
Input
Output
Edge Cases & Gotchas
- Task state in database - Tasks that read Django model state at queue time vs. execution time get stale data. Pass IDs not objects - re-fetch the model in the task.
- Soft time limit - Set soft_time_limit and time_limit on tasks that could run indefinitely. Soft raises SoftTimeLimitExceeded (caught); hard kills the worker process.
- Beat duplication - Running two Beat instances schedules tasks twice. Beat must be a singleton. Use Redbeat (Redis-based lock) in multi-instance deployments.
- Result backend memory - Task results accumulate in Redis forever by default. Set result_expires = 3600 (1 hour) to auto-expire results.
When to Break These Rules
- Use RQ (Redis Queue) instead of Celery for simple background tasks in smaller projects - far simpler setup with no canvas or beat complexity.
- Use Temporal.io for complex multi-step workflows with human approval steps, long-running processes (days/weeks), and durable execution guarantees.
Reference Files
https://docs.celeryq.dev/https://flower.readthedocs.io/