Task Queue + Celery

What is a Task Queue?
A task queue is a mechanism used in software architecture to offload time-consuming or resource-intensive operations from the main application thread to a separate background process. Instead of executing a task immediately and blocking the user interface or application's responsiveness, the task is placed into a queue. A 'worker' process then picks up tasks from this queue and executes them asynchronously.

Key benefits of using a Task Queue:
- Improved Responsiveness: The main application can respond to user requests much faster, as it doesn't wait for long operations to complete.
- Scalability: You can easily add more workers to process tasks concurrently, distributing the load and handling spikes in demand.
- Reliability: Tasks can be retried automatically if they fail, ensuring eventual completion. Tasks can also persist even if the application restarts.
- Decoupling: The application that produces tasks is decoupled from the workers that consume them, leading to a more modular and maintainable system.

Common Use Cases:
- Sending email notifications
- Image processing (resizing, watermarking)
- Generating complex reports or data exports
- Calling third-party APIs
- Executing scheduled jobs (e.g., daily data synchronization)

What is Celery?
Celery is a powerful, distributed task queue system for Python. It is designed to be highly reliable and scalable, processing vast numbers of messages while providing operations with the tools to maintain the system.

Core Components of Celery:
1. Celery Client (Producer): This is your main application code (e.g., a Django or Flask web app) that defines tasks and dispatches them to the task queue. It doesn't execute the task directly.
2. Broker (Message Transport): This acts as an intermediary between the client and the workers. When a client dispatches a task, it sends a message to the broker. Workers then fetch tasks from the broker. Popular brokers include Redis, RabbitMQ, and Amazon SQS.
3. Celery Workers (Consumer): These are separate processes that run continuously, listening to the broker for new tasks. When a task message arrives, a worker picks it up, executes the defined function, and optionally returns a result.
4. Backend (Result Store - Optional): If you need to retrieve the results of a task later, Celery can store them in a backend. This can be the same as your broker (e.g., Redis) or a separate database (e.g., PostgreSQL, MongoDB).

How Celery Works (Simplified Flow):
1. Your Python application calls a Celery task (e.g., `my_task.delay(args)`).
2. Celery serializes the task details (function name, arguments) and sends them as a message to the configured Broker.
3. A Celery Worker process, which is constantly polling the broker, receives the task message.
4. The worker deserializes the message, finds the corresponding task function, and executes it.
5. (Optional) If the task is configured to return a result, the worker sends the result to the Backend.
6. Your original application can then query the backend to retrieve the task's result if needed.

Celery provides features like task scheduling (with Celery Beat), periodic tasks, retries, rate limiting, and monitoring, making it a robust solution for managing background jobs.

 1. Prerequisites
 You need Python installed.
 You also need a message broker, like Redis. You can download and run Redis locally.

 2. Installation
 pip install celery redis

 3. Create the Celery application and define tasks (save as 'celery_app.py')

 celery_app.py
from celery import Celery
import time

 Configure Celery
 'my_tasks' is the name of our Celery application (module name)
 broker='redis://localhost:6379/0' specifies Redis on localhost port 6379, database 0
 backend='redis://localhost:6379/0' specifies where to store task results
app = Celery(
    'my_tasks',
    broker='redis://localhost:6379/0',
    backend='redis://localhost:6379/0'
)

 Define a simple task
@app.task
def add(x, y):
    print(f"Worker: Executing task add({x}, {y})...")
    time.sleep(5)   Simulate a long-running task that takes 5 seconds
    result = x + y
    print(f"Worker: Task add({x}, {y}) completed. Result: {result}")
    return result

 4. Create a producer script to send tasks (save as 'producer.py')

 producer.py
from celery_app import add

print("Producer: Sending task...")

 Call the task asynchronously using .delay()
 This sends the task to the broker immediately and returns an AsyncResult object
result_async = add.delay(4, 5)

print(f"Producer: Task sent with ID: {result_async.id}")
print("Producer: Application continues immediately (non-blocking)...")

 In a real application, you might do other work here or just let the script exit.
 For demonstration, we will block to get the result.
 .get() will block until the result is available or a timeout occurs.
try:
    print("Producer: Waiting for task result (this is blocking for demonstration, up to 10 seconds)...")
    result = result_async.get(timeout=10)  Blocks for up to 10 seconds
    print(f"Producer: Received result: {result}")
except Exception as e:
    print(f"Producer: Error getting result: {e}")

print("Producer: Script finished.")

 5. How to run this example

 Step 5.1: Start your Redis server
 Open a terminal and run:
 redis-server

 Step 5.2: Start the Celery worker
 Open a NEW terminal (keep Redis running in another) in the same directory as celery_app.py
 and run:
 celery -A celery_app worker --loglevel=info
 You should see output indicating the worker is ready and listening for tasks.

 Step 5.3: Run the producer script
 Open a THIRD terminal in the same directory as producer.py and run:
 python producer.py

 Expected Output:
 In the 'producer.py' terminal:
 Producer: Sending task...
 Producer: Task sent with ID: <some-uuid>
 Producer: Application continues immediately (non-blocking)...
 Producer: Waiting for task result (this is blocking for demonstration, up to 10 seconds)...
 (after 5 seconds)
 Producer: Received result: 9
 Producer: Script finished.

 In the 'celery worker' terminal:
 Worker: Executing task add(4, 5)...
 (after 5 seconds)
 Worker: Task add(4, 5) completed. Result: 9