Week 5: Background Processing Control Plane
What We Are Building
InfraWatch AsyncOps — a production background-processing console that ships real work off the request path:
Celery execution layer — Multi-queue routing, retries, worker scaling, Flower visibility
Async task catalog — Metrics collection, batch processing, notifications, reports, maintenance
Cron scheduler — User-defined jobs with priority and manual execution
Workflow orchestration — DAG dependencies, conditional nodes, chord/chain patterns
Monitoring plane — WebSocket streaming, Prometheus metrics, queue depth alerts
Operator console — Datadog/Celery-style dark dashboard with live controls
Core Concept: Request Path vs Worker Path
User-facing APIs must return in milliseconds. Anything that touches disks, networks, or aggregation belongs on a worker path — queued, retried, and observed separately. Stripe and GitHub run variations of this split: HTTP acknowledges intent; workers guarantee completion.
The non-obvious detail: queue naming is a capacity contract, not decoration. A
metrics.highqueue with dedicated workers prevents report generation from starving health checks — the same pattern AWS SQS priority queues and Sidekiq weighted queues use.
Where This Fits
Metrics Collection telemetry source
│
Background Processing ◀ this lesson
│
Alert Management evaluates completed work & thresholds
│
Notifications delivers outcomes
AsyncOps is the execution backbone between raw metrics and human response. Alerts assume tasks complete reliably and emit measurable latency.
Component Architecture
API — FastAPI enqueues tasks, exposes stats, and streams WebSocket updates.
Broker — Redis holds queues and result backends with isolated DB indexes.
Workers — Celery consumers per queue family with autoretry and progress metadata.
Scheduler — Croniter loop plus Celery Beat for platform housekeeping.
Orchestrator — DAG engine resolves dependencies before dispatching child tasks.
Console — React operator UI with queue pause/resume, retry/cancel, workflow triggers.
