Gyaan

Background Tasks

intermediate fastapi background celery production

Sometimes we want to do work that doesn’t need to block the response. Sending a welcome email after signup is the textbook case — the user shouldn’t wait 2 seconds staring at a spinner just because our SMTP server is slow.

FastAPI ships BackgroundTasks for exactly this. In simple language: “I’ll send the response now, but please run this function right after.”

The pattern

We declare a BackgroundTasks parameter, attach functions to it, and return. FastAPI sends the response, then runs the queued functions.

from fastapi import BackgroundTasks, FastAPI

app = FastAPI()

def send_welcome_email(email: str, name: str):
    # slow, blocking, that's fine — runs after response
    smtp.send(to=email, subject="Welcome!", body=f"Hi {name}")

@app.post("/signup")
def signup(email: str, name: str, tasks: BackgroundTasks):
    user = create_user(email, name)
    tasks.add_task(send_welcome_email, email, name)
    return {"user_id": user.id}   # response goes out NOW

The user sees their response in 50ms. The email sends in the background.

Background task timeline
t=0ms   POST /signup arrives
t=20ms  create_user() — DB write
t=22ms  add_task(send_email) — just queues, doesn't run
t=25ms  return response ─► client gets 200 OK
t=25ms  send_welcome_email() starts running
t=2025  email finally sent (client doesn't care)

Sync or async tasks both work

async def push_to_webhook(url: str, payload: dict):
    async with httpx.AsyncClient() as c:
        await c.post(url, json=payload)

@app.post("/event")
def event(tasks: BackgroundTasks, payload: dict):
    save(payload)
    tasks.add_task(push_to_webhook, "https://hooks.slack.com/...", payload)
    return {"ok": True}

Sync tasks run in the thread pool. Async tasks run on the event loop. Same as routes.

Multiple tasks

We can queue several. They run in order.

@app.post("/order")
def place_order(order: Order, tasks: BackgroundTasks):
    saved = save(order)
    tasks.add_task(send_confirmation_email, order.email)
    tasks.add_task(notify_warehouse, saved.id)
    tasks.add_task(update_analytics, "order_placed", saved.id)
    return saved

The big limitations

BackgroundTasks is a great fit for cheap fire-and-forget work, but it has real limits.

  1. Runs in the same process. If our server crashes between sending the response and finishing the task, the task is lost. No retries.
  2. No persistence. Restart the server, in-flight tasks die.
  3. No scheduling. Can’t say “run this in 1 hour”.
  4. Uses the same worker. Heavy CPU tasks will eat into request-handling capacity.
  5. No visibility. No dashboard, no logs of past tasks, no way to retry a failed task.

When to reach for Celery (or similar)

Use Celery / RQ / Dramatiq / arq when we need:

  • Reliability — tasks survive crashes (backed by Redis/RabbitMQ).
  • Retries — failed task automatically retries with backoff.
  • Scheduling — “send this email tomorrow morning”.
  • Heavy work — image processing, ML inference, big PDFs. Don’t bog down API workers.
  • Visibility — Flower dashboard, retry counts, success/failure metrics.
  • Scale — separate worker pool we can grow independently.

Mental model: BackgroundTasks for “after the response, quickly”. A task queue for “anything important”.

Quick Celery sketch

# tasks.py
from celery import Celery

celery = Celery("app", broker="redis://localhost:6379/0")

@celery.task(bind=True, max_retries=3)
def send_welcome_email(self, email: str, name: str):
    try:
        smtp.send(to=email, subject="Welcome", body=f"Hi {name}")
    except SMTPException as e:
        raise self.retry(exc=e, countdown=60)

# main.py
from tasks import send_welcome_email

@app.post("/signup")
def signup(email: str, name: str):
    user = create_user(email, name)
    send_welcome_email.delay(email, name)   # pushes to Redis, worker picks up
    return {"user_id": user.id}

Run a separate worker process: celery -A tasks worker --loglevel=info. Now restart the API all we want — pending tasks survive in Redis.

arq — the asyncio-native alternative

If our whole stack is async, arq feels more natural than Celery. Same idea, Redis-backed, but built on asyncio from day one.

Interview cheat sheet

  • BackgroundTasks runs after the response, in the same process.
  • Perfect for fast fire-and-forget work (emails, webhooks, cache invalidation).
  • Loses tasks on crash. No retries. No persistence.
  • For anything important, use Celery / RQ / arq with a proper broker (Redis/RabbitMQ).
  • Rule of thumb: if losing the task would be a bug, use a real queue.

References