What We’re Building Today

By the end of this lesson you will have a working, production-grade performance testing platform — not a toy, not a demo with mock numbers. Here is what gets built:

• A Locust-powered test suite with five distinct test profiles, each answering a different question about your system

• A FastAPI backend that orchestrates test runs, polls live metrics every two seconds, computes latency percentiles, evaluates SLO thresholds, and streams everything over WebSocket

• A React dashboard — styled after tools like Grafana and k6 Cloud — that shows live RPS charts, P50/P95/P99 latency curves, error rates, and a real-time SLO pass/fail verdict

• A SQLite store for run history so you can compare test results across deployments

The full project runs with a single command, with or without Docker.

Read more

- A Flask API protected by JWT, role checks, rate limiting, and security headers

- One scan pipeline combining SAST, DAST, dependency, and compliance checks

- A React dashboard that refreshes from every real scan

- Repeatable local and Docker workflows with automated verification

This closes Week 20’s testing loop: earlier tests asked whether features work; today we ask whether an attacker can make them work **against us**.

The core idea: evidence, not a green badge

A security scanner is an evidence pipeline. Source code, dependencies, and a running API are different evidence sources, so one tool cannot honestly cover all three. Bandit inspects Python syntax, `pip-audit` checks published advisories, and DAST sends hostile HTTP requests to the live service.

The non-obvious lesson is that “zero findings” is not automatically good. It may mean the scanner could not reach its target, parsed no output, or ran with the wrong configuration. Production systems record scan type, timestamp, tool, severity, evidence, and failures so operators can distinguish “clean” from “not measured.”

Read more

• Quality Gate engine — enforces pass/fail thresholds before deployments

• Code quality metrics — static analysis for complexity, duplication, and coverage

• Technical debt tracker — converts violations to engineer-hours and trends over time

• Performance benchmark harness — catches latency regressions automatically

• Quality Dashboard — a SonarQube-inspired React UI showing all metrics in real-time

Where This Fits in the Overall System

[Source Code] ──> [Test Automation Pipeline (Day 135)]
                               |
                               v
                  [Quality Gate Engine]  <── TODAY
                               |
           +──────────────+──────────────+
           v               v              v
   [Code Metrics]  [Debt Tracker]  [Perf Benchmarks]
           +──────────────+──────────────+
                               |
                               v
                     [Quality Dashboard UI]
                               |
                               v
                  [Deployment Decision: PASS / FAIL]

This module sits between CI and deployment — it is the enforcer. Tomorrow (Day 137) we add security testing on top of this gate.

Component Architecture

Read more

• A CI/CD-ready test automation pipeline — one orchestrator that drives every test layer in the right order

• A parallel test execution engine — three suites running simultaneously, cutting wall-clock time by 60–70%

• A centralised reporting dashboard — real-time pass/fail/skip counters, duration trends, and per-test failure detail

• A test data management layer — deterministic fixture factories, SQLite snapshots, isolated teardown

• A continuous testing loop — a file-watcher that re-runs only the tests affected by your last save

Why Any of This Matters

Netflix runs roughly 10,000 automated tests before a single commit reaches production. Google’s CI infrastructure runs more than 150 million test executions every day across hermetic sandboxes. The gap between a 12-minute test suite and a 2-hour one is almost never better hardware. It is pipeline architecture — what runs in parallel, what is cached, and what never runs at all when nothing relevant changed.

When a pipeline is built correctly, it becomes a living contract between the codebase and its expected behaviour. Break that contract and the entire team knows within minutes, not at 3 AM after a bad deployment.

Component Architecture

The pipeline is five cooperating layers:

Layer File What it does Orchestrator pipeline.py Reads config, resolves run order, fans work out to the thread pool Executor Pool executor.py One subprocess per suite; each gets its own SQLite schema Reporter reporter.py Parses JUnit XML, computes flake rates, writes pipeline_results.json Data Manager data_manager.py Seeded fixture factory, snapshot files, clean teardown per worker Watcher watcher.py watchdog FS events, 500 ms debounce, import-graph-based selection

The React frontend connects to a FastAPI server and receives live events over Server-Sent Events (SSE). There is no polling, no WebSocket handshake — just a plain HTTP stream that the browser reconnects to automatically if it drops.

Control Flow and Data Flow

Here is the full path from a source file change to a result on screen:

Source Change
     |
     v
[ Watcher ]  -- debounce 500 ms -->  [ Change Detector ]
                                              |
                                   [ Affected Test Map ]
                                    (reverse dep lookup)
                                              |
                                       [ Orchestrator ]
                                        (pipeline.py)
                                    /         |         \
                               Unit       Integration    E2E
                             Worker 0      Worker 1    Worker 2
                                    \         |         /
                                   [ Result Aggregator ]
                                              |
                                  [ JUnit XML  +  JSON ]
                                              |
                                   [ FastAPI SSE stream ]
                                              |
                                    [ React Dashboard ]

The orchestrator submits all three suites to a ThreadPoolExecutor simultaneously, then iterates as_completed() — it processes results in whichever order the workers finish, never blocking on the slowest one.

Parallel Execution — The Right Mental Model

Most engineers run pytest -n auto and consider the problem solved. The real issue is test isolation at the data layer. If two parallel workers both seed a users table with fixed IDs into the same database, they corrupt each other’s data midway through and produce failures that only occur when tests run simultaneously — the worst kind to debug.

The approach here gives each worker its own SQLite file, named by worker ID. The file is created at setup and deleted at teardown. Nothing leaks between runs.

# pseudo-code — full implementation is in data_manager.py
class WorkerDataManager:
    def __init__(self, worker_id: int):
        self.db_path = reports_dir / f"worker_{worker_id}.db"

    def setup(self):
        # Start from a clean file every time
        if self.db_path.exists():
            self.db_path.unlink()
        self._run_migrations()
        self._seed_fixtures()

    def teardown(self):
        self._conn.close()
        self.db_path.unlink()   # no files left behind

This eliminates an entire class of timing-sensitive failures. Worker count scales with CPU cores. On CI you pin it to nproc or whatever the machine’s core count is.

Continuous Testing — Smart Selection

Running all 500 tests on every keystroke is expensive and slow. The watcher uses import graph analysis — when services/billing.py changes, only tests whose import chain touches billing get re-run. This is how Jest’s --watch mode and the pytest-testmon plugin work internally.

# pseudo-code
changed_modules = detect_changed_files(event)
affected_tests  = import_graph.reverse_deps(changed_modules)
runner.execute(affected_tests, mode="fast_feedback")

The full suite still runs on every git push through the CI gate. The watcher is your local feedback loop — under three seconds for a change in a leaf module, which keeps the flow state intact while you are working.

Test Reporting — What Good Looks Like

Raw pass/fail counts are nearly useless. Production-grade reporting surfaces four things:

Flake rate per test — the ratio of failures to total runs over the last N executions. Anything above 2% gets auto-quarantined. A test that fails one time in twenty is not a passing test.

Duration trend — P50 and P95 duration over the last seven days. A test that was 200 ms last week and is now 1.8 s has a regression somewhere, and it will cause timeouts in CI before anyone notices.

First-failure lines — pulled directly from the JUnit XML stacktrace. Nobody should have to click into a separate log viewer to see why a test failed.

Coverage delta — did this run improve or regress line coverage? A green pipeline that quietly drops coverage from 84% to 61% is not a green pipeline.

The dashboard here produces exactly what CircleCI’s Test Insights tab and Buildkite’s test analytics show for their enterprise customers.

Test Data Management — Snapshots and Factories

Test data breaks in two opposite directions. Tests that are too tightly coupled to specific row IDs fail whenever the seed data changes. Tests that use fully random data cannot reproduce the same failure twice.

The approach that splits the difference is a factory function seeded with a deterministic PRNG. The seed is derived from the test’s node ID — which means the same test always generates exactly the same fake data, and two different tests never collide.

# pseudo-code
class TestDataFactory:
    def user(self, **overrides):
        faker = Faker()
        faker.seed_instance(hash(self.test_node_id) % 2**32)
        base = {"name": faker.name(), "email": faker.email()}
        return {**base, **overrides}

Snapshot testing extends this one step further. After the first passing run, the factory serialises the generated dataset to fixtures/snapshots/<test_id>.json. On subsequent runs, if that file exists, it is replayed exactly — no randomness, maximum reproducibility, and no brittle hardcoded fixtures in your repository.

Project Structure

Github Link:

https://github.com/sysdr/infrawatch-fullstack-p/tree/main/day135/infra-dashboard

infra-dashboard/
├── backend/
│   ├── test_pipeline/
│   │   ├── pipeline.py          # orchestrator — fans out to worker pool
│   │   ├── executor.py          # subprocess runner per suite
│   │   ├── reporter.py          # JUnit XML → JSON aggregator + flake calc
│   │   ├── data_manager.py      # fixture factory + SQLite isolation
│   │   └── watcher.py           # FS watcher → debounce → selective run
│   ├── api/
│   │   └── pipeline_routes.py   # SSE /run, /results, /history endpoints
│   ├── tests/
│   │   ├── unit/
│   │   ├── integration/
│   │   └── e2e/
│   ├── main.py
│   └── conftest.py
├── frontend/
│   └── src/
│       ├── pages/TestDashboard.jsx
│       └── api/pipelineApi.js
├── start.sh
├── stop.sh
└── docker-compose.yml

Prerequisites

Before running anything, confirm these are installed:

python3 --version    # Python 3.11 or 3.12
node --version       # v20.x
npm --version        # 10.x
docker --version     # optional, for containerised run

Part 1 — Build and Run (Without Docker)

Step 1: Run the build script

chmod +x start.sh
./start.sh

This creates the entire project from scratch, installs all Python dependencies, runs all three test suites, executes the pipeline smoke-test, and starts both backend and frontend.

What you should see:

[BUILD] Creating project structure...
[BUILD] Writing data_manager.py...
[BUILD] Writing reporter.py...
[BUILD] Writing executor.py...
[BUILD] Writing pipeline.py...
[BUILD] Writing watcher.py...
[BUILD] Writing pipeline_routes.py...
[BUILD] Writing main.py...
[BUILD] Writing tests...
[BUILD] Writing frontend files...
[BUILD] Verifying file structure...
  ✓ infra-dashboard/backend/test_pipeline/pipeline.py
  ✓ infra-dashboard/backend/test_pipeline/executor.py
  ✓ infra-dashboard/backend/test_pipeline/reporter.py
  ✓ infra-dashboard/backend/test_pipeline/data_manager.py
  ✓ infra-dashboard/backend/test_pipeline/watcher.py
  ✓ infra-dashboard/backend/api/pipeline_routes.py
  ✓ infra-dashboard/backend/main.py
  ... 21 files total ✓
[BUILD] Python dependencies installed ✓

Step 2: Inspect what was created

find infra-dashboard -type f -name "*.py" | sort

You should see thirteen Python files across test_pipeline/, api/, tests/, and the root.

Part 2 — Manual Backend Setup (Alternative)

If you want to set up without the build script, or if you are integrating this into an existing project:

cd infra-dashboard/backend
python3 -m venv venv
source venv/bin/activate          # Windows: venv\Scripts\activate
pip install -r requirements.txt

Part 3 — Running the Tests

Run each suite individually first to make sure every layer is working independently before you fire the parallel pipeline.

Unit tests

These test the WorkerDataManager class directly — schema creation, fixture seeding, factory determinism, and clean teardown.

cd infra-dashboard/backend
PYTHONPATH=$(pwd) python -m pytest tests/unit/ -v --tb=short

Expected output:

tests/unit/test_data_manager.py::test_setup_creates_tables          PASSED
tests/unit/test_data_manager.py::test_fixture_seeds_users           PASSED
tests/unit/test_data_manager.py::test_fixture_seeds_tasks           PASSED
tests/unit/test_data_manager.py::test_factory_user_deterministic    PASSED
tests/unit/test_data_manager.py::test_factory_user_override         PASSED
tests/unit/test_data_manager.py::test_factory_task_deterministic    PASSED
tests/unit/test_data_manager.py::test_query_returns_list            PASSED
tests/unit/test_data_manager.py::test_teardown_removes_db           PASSED
tests/unit/test_executor_pipeline.py::test_suite_result_passed_when_zero   PASSED
tests/unit/test_executor_pipeline.py::test_suite_result_failed_when_nonzero PASSED
tests/unit/test_executor_pipeline.py::test_suite_result_stores_attributes   PASSED
tests/unit/test_executor_pipeline.py::test_suite_result_xml_path_reference  PASSED

12 passed in 0.3s

Integration tests

These test the Reporter class — XML parsing, report building, flake rate calculation, quarantine flag logic, and history persistence.

PYTHONPATH=$(pwd) python -m pytest tests/integration/ -v --tb=short

Expected output:

tests/integration/test_reporter.py::test_parse_junit_xml_counts                    PASSED
tests/integration/test_reporter.py::test_parse_junit_xml_statuses                  PASSED
tests/integration/test_reporter.py::test_reporter_builds_report                    PASSED
tests/integration/test_reporter.py::test_reporter_flake_rate_zero_for_new_test     PASSED
tests/integration/test_reporter.py::test_reporter_report_has_timestamp             PASSED
tests/integration/test_reporter.py::test_reporter_quarantine_flag                  PASSED
tests/integration/test_reporter.py::test_reporter_history_persisted                PASSED
tests/integration/test_reporter.py::test_parse_missing_xml_returns_empty           PASSED

8 passed in 0.6s

E2E tests

These test the FastAPI layer via TestClient — all three API endpoints plus CORS header behaviour.

PYTHONPATH=$(pwd) python -m pytest tests/e2e/ -v --tb=short

Expected output:

tests/e2e/test_api_pipeline.py::test_health_endpoint        PASSED
tests/e2e/test_api_pipeline.py::test_results_404_before_run PASSED
tests/e2e/test_api_pipeline.py::test_history_returns_list   PASSED
tests/e2e/test_api_pipeline.py::test_history_run_structure  PASSED
tests/e2e/test_api_pipeline.py::test_cors_headers           PASSED

5 passed in 1.2s

All suites at once

PYTHONPATH=$(pwd) python -m pytest tests/ -v --tb=short

25 passed in 0.96s

Part 4 — Pipeline Smoke Test

This is the real proof of parallel execution. Run it and watch all three suites start before any of them finish.

cd infra-dashboard/backend
PYTHONPATH=$(pwd) python -c "
import sys; sys.path.insert(0, '.')
from test_pipeline import run_pipeline

def cb(e): print(f'  [{e[\"event\"]}] {e.get(\"suite\",\"pipeline\")}')
report = run_pipeline(progress_cb=cb)
s = report['summary']
print(f'passed={s[\"passed\"]} failed={s[\"failed\"]} total={s[\"total\"]}')
print(f'pipeline_duration={report[\"pipeline_duration\"]}s')
"

Expected output:

  [started] unit
  [started] integration
  [started] e2e
  [suite_done] integration
  [suite_done] unit
  [suite_done] e2e
  [pipeline_done] pipeline
passed=21 failed=0 total=21
pipeline_duration=2.26s

Notice that integration finishes before unit even though unit started first — the suites are genuinely running simultaneously. The pipeline wall time (~2.3s) is the duration of the single slowest suite, not the sum of all three (~5.4s sequential).

Part 5 — Starting the API Server

cd infra-dashboard/backend
PYTHONPATH=$(pwd) uvicorn main:app --host 0.0.0.0 --port 8000 --reload

Expected:

INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000

Verify all three API endpoints

# Health check
curl http://localhost:8000/health
# {"status":"ok","service":"test-pipeline"}

# Latest pipeline results (available after first SSE run)
curl http://localhost:8000/api/pipeline/results

# Run history from SQLite
curl http://localhost:8000/api/pipeline/history
# {"runs": [...]}

# Interactive API docs
open http://localhost:8000/docs

Part 6 — Starting the Frontend

cd infra-dashboard/frontend
npm install --legacy-peer-deps
npm start

Expected:

Compiled successfully!
Local: http://localhost:3000

Part 7 — Live Dashboard Walkthrough

Open

http://localhost:3000

. The dashboard has four tabs across the top.

Overview tab — this is what you land on. Four metric cards across the top show Passed, Failed, Skipped, and Total. Below them is a bar chart breaking results down per suite. Below the chart is the live event log.

Running the pipeline:

1. Click the green Run Pipeline button in the header

2. The live log immediately starts showing events as they stream in from the SSE endpoint:

[10:32:14] started: unit[10:32:14] started: integration[10:32:14] started: e2e[10:32:16] suite_done: integration[10:32:17] suite_done: unit[10:32:18] suite_done: e2e[10:32:18] pipeline_done:

1. The metric cards update when pipeline_done arrives

2. The bar chart redraws with real numbers

Suites tab — three cards, one per suite. Each shows a PASS or FAIL badge, the test count, and the execution time. While the pipeline is running, the card says RUNNING in green.

Tests tab — a full table of every test with its suite, status badge, duration in seconds, flake rate percentage, and a warning icon if the test is quarantined (flake rate above 5%). Quarantined rows have an amber background.

History tab — every test result from every previous run, pulled from run_history.db. This is where you would eventually build duration trend charts.

Part 8 — Docker Run

If you prefer containers, the same build script handles everything with a flag:

./start.sh --docker

Or step by step:

docker-compose build --no-cache
docker-compose up -d

Expected:

[+] Building backend  ... done
[+] Building frontend ... done
[+] Running 2/2
 ✔ Container backend   Started
 ✔ Container frontend  Started

Verify the containers are healthy:

docker-compose ps
# NAME       STATUS    PORTS
# backend    Up        0.0.0.0:8000->8000/tcp
# frontend   Up        0.0.0.0:3000->80/tcp

curl http://localhost:8000/health
# {"status":"ok","service":"test-pipeline"}

Stream backend logs (includes live test output from each pipeline run):

docker-compose logs -f backend

Stop everything:

./stop.sh
# or
docker-compose down

Part 9 — Testing the Watcher

Open a second terminal and start the watcher:

cd infra-dashboard/backend
source ../../venv/bin/activate
python -c "
import sys; sys.path.insert(0, '.')
import time
from test_pipeline.watcher import ContinuousTestRunner

runner = ContinuousTestRunner()
runner.start()
print('Watcher running — edit any .py file in tests/ to trigger re-run')
try:
    while True: time.sleep(1)
except KeyboardInterrupt:
    runner.stop()
"

In your original terminal, touch any test file:

touch infra-dashboard/backend/tests/unit/test_data_manager.py

Expected watcher output (within 500 ms):

[WATCHER] Watching .../backend/tests
[WATCHER] Change detected: .../tests/unit/test_data_manager.py — triggering selective run
[WATCHER] Running pipeline...
[WATCHER] Done — 21 passed, 0 failed

The 500 ms debounce means rapid successive saves (like auto-format on save) only trigger one run, not ten.

Part 10 — Verifying the Reports

After at least one pipeline run, check the output files:

ls infra-dashboard/backend/reports/

junit_unit.xml
junit_integration.xml
junit_e2e.xml
pipeline_results.json
run_history.db

Inspect the JSON report directly:

cat infra-dashboard/backend/reports/pipeline_results.json | python3 -m json.tool | head -40

Expected structure:

{
  "timestamp": "2025-...",
  "summary": {
    "passed": 21,
    "failed": 0,
    "skipped": 0,
    "error": 0,
    "total": 21
  },
  "suites": ["unit", "integration", "e2e"],
  "tests": [
    {
      "test_id": "unit::tests.unit.test_data_manager.test_setup_creates_tables",
      "suite": "unit",
      "status": "passed",
      "duration": 0.03,
      "flake_rate": 0.0,
      "quarantined": false
    }
  ],
  "pipeline_duration": 2.26
}

Success Criteria

Work through this checklist before moving on to Day 136. Every item should be a confirmed yes.

What to check How to check it 12 unit tests pass pytest tests/unit/ -v shows 12 passed 8 integration tests pass pytest tests/integration/ -v shows 8 passed 5 E2E tests pass pytest tests/e2e/ -v shows 5 passed All 25 tests pass at once pytest tests/ -v shows 25 passed Pipeline runs in parallel Smoke-test wall time < sum of individual suite times Schema isolation holds No worker_*.db files remain after pipeline completes Fixtures are deterministic Run WorkerDataManager twice with same node ID → same data SSE stream works Dashboard live log shows started + suite_done + pipeline_done Reports are written reports/pipeline_results.json and run_history.db both exist Watcher triggers correctly touch a test file → pipeline re-runs within 1 second Docker build succeeds docker-compose build exits with code 0

Troubleshooting

ModuleNotFoundError: No module named 'test_pipeline'

The backend root is not on the Python path. Fix it with:

export PYTHONPATH=$(pwd)/infra-dashboard/backend:$PYTHONPATH

The build script sets this automatically. If you are running commands manually, you need it every session.

SSE stream closes immediately without events

The backend is not running when you click Run Pipeline. Check:

curl http://localhost:8000/health

If that fails, start the backend first (Part 5 above).

Frontend shows a blank page

Open the browser console. If you see a CORS error, REACT_APP_API_URL is pointing somewhere unexpected. Unset it or set it explicitly:

REACT_APP_API_URL=http://localhost:8000 npm start

Port already in use

./stop.sh
lsof -ti:8000 | xargs kill -9
lsof -ti:3000 | xargs kill -9

Key Insights for When You Design Systems

Pipeline as code. pipeline.py is not a dev tool. It is a production artefact. Treat it the same way you treat application code — version it, code-review it, test the pipeline itself.

Schema isolation is cheaper than database isolation. Spinning up a separate database container per worker costs time and memory. Creating a separate SQLite file per worker takes microseconds and zero extra memory. Use the cheapest isolation that is still actually isolated.

Quarantine flaky tests automatically. A test that fails intermittently is not a passing test. It is a test that is lying to you about half the time. Flag it, exclude it from the gate, and fix it in a separate branch.

SSE over WebSockets for one-way dashboards. If the server is the only one pushing data, WebSockets adds unnecessary handshake overhead. SSE is a plain HTTP response with a persistent connection, has native browser reconnection logic, and passes through every proxy and load balancer that HTTP passes through.

Assignment

Task — Extend the flake detection logic with quarantine events.

This extends everything you built today. You are adding three things: persistent flake history, automatic quarantine detection, and a visual signal in the dashboard.

Steps:

1. Open infra-dashboard/backend/test_pipeline/reporter.py. In the ingest() method, confirm the run_timestamp field is being written to test_runs. It already is — take a moment to read the insert statement and understand how the history builds up.

2. Add a method get_quarantined_tests(threshold=5.0) to the Reporter class. It should query run_history.db for tests where the failure ratio over all recorded runs exceeds the threshold:

SELECT test_id,
       COUNT(*) as total,
       SUM(CASE WHEN status='failed' THEN 1 ELSE 0 END) as fails
FROM test_runs
GROUP BY test_id
HAVING (fails * 1.0 / total) > 0.05

3. In api/pipeline_routes.py, after the pipeline_done event is yielded in the SSE generator, call get_quarantined_tests() and yield a second event:

{"event": "quarantine", "tests": [list of quarantined test_ids]}

4. In frontend/src/pages/TestDashboard.jsx, add a listener for quarantine events in the subscribePipeline callback. Store the quarantined list in a new state variable.

5. On the Overview tab, render a new amber-coloured card below the metric cards. It should only appear when the quarantined list is non-empty, and should list each quarantined test ID.

Hint for the dashboard card:

Use a background of #fffbeb with a #d97706 border and heading — matching the amber colour already used on the quarantine warning icons in the Tests table. Consistency matters in a dashboard.

Test File Reference

File Tests What it covers tests/unit/test_data_manager.py 8 WorkerDataManager — schema creation, fixture seeding, deterministic factory, teardown cleanup tests/unit/test_executor_pipeline.py 4 SuiteResult struct — returncode mapping, attribute storage, XML path handling tests/integration/test_reporter.py 8 Reporter — JUnit XML parsing, report building, flake rates, quarantine flag, history persistence tests/e2e/test_api_pipeline.py 5 FastAPI endpoints — /health, /results, /history, CORS headers Total 25 Full stack: data layer → executor → reporter → API

Working Code Demo:

# Layer Tools 1 Unit test suite pytest · React Testing Library 2 Integration test suite FastAPI TestClient · real PostgreSQL 3 End-to-end test suite Playwright · Chromium 4 Visual regression Playwright screenshots · pixel diff 5 Accessibility audit axe-core injected into every E2E run

Why a Testing Pyramid — and Why It Matters at Scale

Google, Netflix, and Shopify all operate on the same principle: the cost of a bug multiplies with how late you catch it. A unit test that runs in 5 ms catches logic errors in milliseconds. A production incident takes hours of engineering time and burns user trust that is very hard to earn back.

The classic Testing Pyramid — many unit tests, fewer integration tests, fewest E2E tests — is not academic doctrine; it is a throughput constraint. If 80% of your coverage lives in fast unit tests, your CI pipeline finishes in 3 minutes. Invert the pyramid and every pull request blocks for 45 minutes.

              /\
             /E2\           few · slow · high-confidence
            /----\
           / Int. \         moderate · sits at the API boundary
          /--------\
         /  Unit   \        many · fast · pure logic
        /------------\

Read more

You built the mobile-first UI with gestures, forms, and accessible dashboards on Day 132. Today you close the loop — validate that every moving part of your PWA actually works end-to-end on a real device or emulated environment. This is the day your mobile product becomes provably shippable.

What We Build Today

By the end of this lesson you will have:

• A PWA Audit Pipeline — Lighthouse-driven automated scoring for installability, performance, and offline readiness

• An Offline Cache Validation layer — Service Worker cache integrity tests that prove your app works with zero network

• A Push Notification Test Harness — backend-triggered push events verified through a Python test runner

• A Mobile Performance Benchmark Dashboard — real-time FCP, TTI, CLS metrics collected and rendered in React

• A complete End-to-End Mobile Test Suite — Playwright automation that simulates a Pixel 7 and iPhone 14 browser session

Where This Fits

In Week 18 you built the backend API and data layers. Week 19 wraps that work with a tested, installable, offline-capable mobile experience. Week 20 (Day 134 onwards) expands this test harness into a full QA strategy — the foundation you lay today becomes the integration test baseline for every feature you build after this.

Component Architecture

Your test infrastructure wraps around the running application like a quality gate. The Python backend exposes a /api/mobile-health endpoint that aggregates five signal types: PWA score, cache status, push delivery, Core Web Vitals, and E2E pass rate. The React dashboard polls this endpoint and renders live status tiles — exactly the kind of internal QA dashboard you would find at teams running large-scale PWAs like Twitter Lite, Flipkart, or Uber Eats Web.

Read more

By the end of today you will have a production-grade mobile UI layer bolted onto your Infrastructure Management dashboard — the same kind of interface that tools like Datadog Mobile, PagerDuty, and Grafana Mobile expose to on-call engineers. Specifically:

• Mobile-first layout system using CSS Grid and Flexbox with responsive breakpoints

• Gesture engine — swipe navigation, pull-to-refresh, long-press context menus

• Mobile-optimized forms with inline validation, autofill hints, and numeric keyboards

• Touch-friendly dashboard cards showing CPU, memory, and incident KPIs

• WCAG 2.1 AA accessibility layer — focus rings, ARIA roles, screen-reader labels

This week’s arc: Day 131 got the app fast (lazy loading, image optimization, caching). Today we make it feel native. Day 133 ties everything together in integration testing.

Where This Fits in the Overall System

[ React PWA Shell ]
      │
      ├── Mobile UI Layer  ◄── YOU ARE HERE
      │      ├── Gesture Engine (touch events)
      │      ├── Responsive Layout Engine
      │      ├── Accessible Form System
      │      └── Mobile Dashboard Cards
      │
      ├── Service Worker (Day 131)
      ├── Push Notification Handler (Day 133)
      └── FastAPI Backend (Week 14–18)

The mobile UI layer sits between the PWA shell and every user-facing component. It intercepts raw browser touch events and normalises them into semantic gestures (swipe, pinch, long-press) that any child component can subscribe to without knowing touch internals.

Read more

Here’s the agenda — sharp and focused:

1. Lazy Loading Engine — defer off-screen images and components until the viewport demands them

2. Image Optimization Pipeline — serve WebP/AVIF with responsive breakpoints via a Python FastAPI microservice

3. Mobile Cache Architecture — a multi-layer cache strategy (Memory → IndexedDB → Network) using Service Workers

4. Performance Monitoring Dashboard — real-time Core Web Vitals collection and visualization

By end of day, you’ll have a React PWA frontend talking to a Python backend with measurable, observable mobile performance — the same pattern Flipkart used when they rebuilt their mobile web as a PWA and saw 70% longer sessions.

Why Mobile Performance Is a Systems Problem, Not a Frontend Problem

Every 100ms of latency costs ~1% in conversions (Amazon, 2023 internal data). On a 3G connection — still 40% of Indian mobile traffic — that’s not hypothetical, it’s revenue bleeding per render.

Mobile performance breaks down across three layers that most tutorials conflate into one:

Network layer — bandwidth is scarce, latency spikes on cell handoff. You optimize here with compression, image format negotiation, and HTTP/2 push. Parse & execute layer — mobile CPUs are 4–6x slower than desktop. JavaScript bundle size and render-blocking resources kill TTI (Time to Interactive). Perceived performance layer — what users feel. Skeleton screens, optimistic UI, and progressive image reveals are the engineering levers here.

Our system targets all three.

Component Architecture

The architecture has three runtime zones:

Client Zone (React PWA): The PerformanceMonitor component wraps the entire app tree. It registers a PerformanceObserver watching largest-contentful-paint, first-input, and layout-shift entries — these are the Core Web Vitals. The LazyLoader HOC (Higher Order Component) uses IntersectionObserver to trigger render only when a component’s root element enters the viewport with a configurable root margin. The ImageOptimizer component negotiates format with the backend by reading navigator.connection.effectiveType and passing it as a query param.

Service Worker Zone: Sits between the network and the React app. Implements a Cache-First strategy for static assets, a Network-First strategy for API calls, and a Stale-While-Revalidate strategy for images. This means users on a flaky connection still get content — it’s served from cache while the SW silently refreshes in the background.

Backend Zone (FastAPI): The /api/images/optimize endpoint receives an image URL, target width, and connection hint. It uses Pillow to resize and convert to WebP/AVIF, caches the result in Redis with a content-hash key, and returns the optimized binary with proper Cache-Control: max-age=31536000, immutable headers.

Read more

InfraWatch Alerts — a production alert management console modeled after PagerDuty and Datadog:

1. Rule engine — Threshold, anomaly, and expression-based rules with templates

2. Evaluation pipeline — Sustained breach detection, deduplication, suppression windows

3. Lifecycle processing — Severity scoring, escalation policies, auto-resolution

4. Query plane — Search, statistics, bulk operations, CSV/JSON export

5. Operator console — Dark-theme dashboard with live WebSocket updates

6. Notification routing — Channel delivery with circuit-breaker protection

Core Concept: Signal vs Noise

Every monitoring platform ingests thousands of metric points per minute. The hard problem is not detection — it is deciding which signals deserve human attention. Datadog and PagerDuty solve this with pending timers (avoid flapping), fingerprint deduplication (one alert per root cause), and suppression windows (maintenance silence).

The non-obvious insight: alert state is a first-class entity, not a side effect of rule evaluation. A rule fires; an alert instance is born with its own lifecycle. Operators acknowledge, escalate, and resolve alerts — not rules.

Where This Fits

Metrics Collection          raw telemetry
        │
Background Processing       scheduled evaluation jobs
        │
Alert Management            ◀ this lesson
        │
Notifications               delivers outcomes to humans

Alerts sit between completed metric pipelines and human response. They assume metrics arrive reliably and workers can evaluate rules on schedule.

Component Architecture

Ingest — Agents POST metric samples; each sample triggers rule evaluation.
Evaluator — Compares values against thresholds or Z-score anomaly models.
Processor — Manages pending→firing transitions, severity scoring, history audit.
Query API — Powers search, statistics, and export for the operator console.
WebSocket — Pushes state changes to the dashboard without polling.

Read more

By the end of this lesson, InfraWatch proactively alerts engineers about system events — even when the browser tab is closed:

• Web Push subscription pipeline — VAPID-authenticated, browser-native delivery

• Notification scheduler — APScheduler with per-user timezone and quiet-hour awareness

• Notification management UI — category-level enable/disable controls per user role

• Personalization engine — role-based defaults, quiet-hour filtering, and snooze support

• Analytics dashboard — delivery rate, CTR, dismiss rate, and subscription health

Why Push Notifications Are an Architectural Decision, Not a Feature

Most developers treat push notifications as a UI checkbox. Engineering teams at PagerDuty and Datadog treat them as a critical reliability contract between the system and the on-call engineer. A notification arriving 4 minutes late during an incident cascade is functionally useless.

The Web Push Protocol (RFC 8030) makes browser notifications independent of your backend’s uptime. Your server doesn’t hold a persistent socket to the browser — it posts an encrypted payload to a vendor-operated push relay (Google’s FCM for Chrome, Mozilla’s autopush for Firefox). That relay maintains the connection and delivers the message when the browser next connects. This is why notifications survive tab closure and network drops.

Component Architecture

Core Concept: VAPID Authentication

VAPID (Voluntary Application Server Identification for Web Push) proves that a notification came from your server, not a spoofed source.

You generate an EC P-256 key pair once. The public key is embedded in your frontend subscription call; the private key signs a short-lived JWT on every push request. The push relay validates that JWT before forwarding the message.

VAPID Private Key  →  signs JWT  →  Authorization header  →  Push Relay validates  →  delivers
VAPID Public Key   →  embedded in PushSubscription  →  tied to your server identity

This is the same elliptic-curve cryptography used in TLS 1.3. The short-lived JWT (24-hour TTL max) means a leaked token has a bounded blast radius.

One rule that trips everyone up: never rotate VAPID keys without first unsubscribing all existing subscribers. Old subscriptions bind to the original public key — they start returning 403 the moment your keys change, silently breaking delivery for every user. Treat VAPID key rotation exactly like rotating a TLS certificate: plan it, notify, and migrate.

Subscription Lifecycle

When a user clicks “Enable Notifications” in InfraWatch, five things happen in sequence:

1. Browser calls Notification.requestPermission() — this requires an explicit user gesture; you cannot trigger it programmatically without a click

2. Service Worker calls pushManager.subscribe({ userVisibleOnly: true, applicationServerKey })

3. Browser contacts the vendor push service and receives a unique push endpoint URL plus two encryption keys: p256dh and auth

4. Frontend POSTs the PushSubscription object to POST /api/subscriptions

5. Backend stores the endpoint and keys in PostgreSQL, tagged to the user ID and role

The subscription object is the user’s mailing address in push-space. It does not belong to your database — it belongs to the browser and the vendor relay. When the relay returns HTTP 410 Gone, that address has been invalidated. Delete it immediately. Stale subscriptions silently inflate your “sent” counter while actual delivery stays at zero.

Notification Scheduling Architecture

InfraWatch uses APScheduler (AsyncIOScheduler) inside FastAPI. Three jobs run on independent schedules:

Job Trigger Purpose heartbeat_check every 60 s Alert digest for triggered monitors daily_summary cron 08:00 user-local Daily system health brief subscription_gc cron 03:00 UTC Remove 410-Gone stale subscriptions

Quiet hours are enforced at the scheduler level, not in JavaScript. If it is 02:00 in the user’s configured timezone, the job skips that user entirely. Client-side suppression is unreliable — the tab might not even be open.

The scheduler must use AsyncIOScheduler, not BackgroundScheduler. The latter runs in a separate thread and cannot await async SQLAlchemy session methods. Mixing them causes event loop conflicts that are difficult to reproduce and nearly impossible to debug under load.

The Service Worker’s Role

The Service Worker is a background JavaScript thread that persists independently of your React app. It registers a push event listener that fires whenever the relay delivers a message.

self.addEventListener('push', event => {
    const data = event.data.json()
    event.waitUntil(
        self.registration.showNotification(data.title, {
            body: data.body,
            icon: data.icon,
            data: { url: data.url, notificationId: data.id }
        })
    )
})

event.waitUntil() is not optional — without it, the browser terminates the Service Worker before showNotification completes and the notification never appears. The notificationclick handler uses clients.openWindow() to navigate the user to the relevant dashboard panel, using the URL embedded in the notification payload.

The Service Worker file must live at public/sw.js so Vite serves it at the root scope /sw.js. A worker registered at /src/sw.js controls only the /src/ scope and will miss every push event.

Personalization: More Than “Turn On/Off”

InfraWatch segments preferences by category: Alerts, Team Updates, Daily Digest, Security Events. Each has independent enable/disable and threshold settings in the notification_preferences table.

Role-based defaults matter: a DevOps engineer defaults to all alert categories on; a product manager defaults to daily digest only. This mirrors PagerDuty’s urgency-tier routing — segmented per on-call rotation, not binary on/off.

Overnight quiet-hour windows (e.g., 22:00–07:00) require a specific branch in the scheduler. The naive check 22 <= hour < 7 is always False because 22 is never less than 7. You need:

if start > end:   # overnight window — e.g., 22:00 to 07:00
    if hour >= start or hour < end:
        continue  # inside quiet window, skip this user
else:
    if start <= hour < end:
        continue

Analytics: What Actually Matters

Four metrics define push system health:

• Delivery rate = delivered / sent (target > 95%; lower values indicate stale subscriptions)

• Click-through rate = clicked / delivered (benchmark: 10–20% for actionable infrastructure alerts)

• Dismiss rate = dismissed / delivered (rising dismiss rate signals low signal-to-noise)

• Subscription churn = unsubscribed this week / active (above 5% means notification fatigue)

The Service Worker reports click and dismiss events to POST /api/notifications/event with the notificationId. The backend timestamps these and links them to the original notification record.

notification_events is a separate table, not columns on notifications. This lets you record multiple events per notification — delivered at T+0, clicked at T+45s — with full timestamps. This is the same pattern PagerDuty uses to calculate acknowledgement latency on incident alerts.

How This Fits InfraWatch

This module connects Week 11’s User & Team Management theme to the alerting system from earlier weeks. The notification subscription lives alongside user profile data. The scheduler reads from the same alert tables built in Weeks 5–7. Analytics feed the same dashboard infrastructure from Week 9.

Architecturally, Day 130 closes the loop: InfraWatch can now reach the engineer rather than waiting for the engineer to open a tab.

Implementation Guide

GitHub Link:-

https://github.com/sysdr/infrawatch/tree/main/day130/infrawatch-day130

Project Structure

Running ./setup.sh creates the full project at ./infrawatch-day130/ and writes every source file as heredoc. No file needs to be created manually.

infrawatch-day130/
├── backend/
│   ├── app/
│   │   ├── api/         notifications.py  subscriptions.py  users.py  vapid.py
│   │   ├── models/      models.py
│   │   ├── schemas/     schemas.py
│   │   └── services/    push_service.py   scheduler_service.py
│   ├── tests/           conftest.py   test_push_notifications.py
│   ├── generate_vapid.py
│   ├── main.py
│   └── requirements.txt
├── frontend/
│   ├── public/          sw.js   manifest.json   icons/
│   └── src/
│       ├── hooks/       usePushNotifications.js
│       ├── pages/       Dashboard.jsx
│       └── services/    api.js
└── docker-compose.yml

Step 1 — VAPID Key Generation

VAPID keys are EC P-256 asymmetric keypairs. Generate them once. In production, store the private key in a secrets manager (Vault, AWS Secrets Manager). The setup script calls generate_vapid.py automatically on first run — but you can run it independently:

cd infrawatch-day130/backend
python generate_vapid.py
# Writes VAPID_PUBLIC_KEY and VAPID_PRIVATE_KEY to .env

The generator uses Python’s cryptography library. The private key is PEM-encoded (required by pywebpush); the public key is URL-safe base64 (required by the browser’s subscribe() call). Never commit the private key to source control.

Verify the key was written:

curl http://localhost:8130/api/vapid/public-key

{ "public_key": "BOM4eVR2T1gNlBMv8UXa2n..." }

If this returns an empty string, the .env file was not populated. Re-run generate_vapid.py.

Step 2 — Database Schema

Four tables, each with a distinct responsibility:

Table Key Fields Purpose push_subscriptions endpoint, p256dh, auth, user_id, is_active One row per browser session per user notification_preferences category flags, quiet_hours_start/end, snoozed_until Per-user delivery rules notifications title, body, category, status, sent_at Every sent notification record notification_events notification_id, event_type, occurred_at Click / dismiss / delivered timestamps

The endpoint column on push_subscriptions carries a unique constraint. Subscription creation always uses an upsert on endpoint — the browser will call subscribe() again after a service worker re-registration, and a blind insert would throw a unique constraint violation every time.

Step 3 — Subscription API

# Pseudo-code: POST /api/subscriptions
existing = query(PushSubscription).filter_by(endpoint=endpoint).first()
if existing:
    existing.is_active = True
    existing.p256dh    = keys.p256dh
    existing.auth      = keys.auth
else:
    db.add(PushSubscription(user_id=user_id, endpoint=endpoint, ...))
# Then upsert role-based default preferences for this user

Create a user and register a subscription:

# Create a user
UID=$(curl -s -X POST http://localhost:8130/api/users \
  -H 'Content-Type: application/json' \
  -d '{"username":"alice","email":"alice@infrawatch.dev","role":"devops","timezone":"UTC"}' \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")
echo "User: $UID"

# Register a subscription for that user
curl -X POST http://localhost:8130/api/subscriptions \
  -H 'Content-Type: application/json' \
  -d "{
    \"user_id\":  \"$UID\",
    \"endpoint\": \"https://fcm.googleapis.com/fcm/send/...\",
    \"keys\": { \"p256dh\": \"BNcR...\", \"auth\": \"tBHI...\" }
  }"

{ "id": "775dbc46...", "user_id": "...", "is_active": true }

Step 4 — Sending with pywebpush

from pywebpush import webpush, WebPushException

webpush(
    subscription_info={
        "endpoint": sub.endpoint,
        "keys": { "p256dh": sub.p256dh, "auth": sub.auth }
    },
    data=json.dumps({ "title": title, "body": body, "url": url, "id": notif_id }),
    vapid_private_key=settings.VAPID_PRIVATE_KEY,
    vapid_claims={
        "sub": settings.VAPID_CLAIMS_EMAIL,
        "aud": extract_audience(sub.endpoint),
    },
)

Three response codes to handle explicitly:

HTTP Status Meaning Action Required 201 Created Accepted by relay Log as delivered 410 Gone Subscription invalidated Delete from DB immediately 429 Too Many Requests Rate limited Log warning, retry with exponential backoff

Send a notification:

NID=$(curl -s -X POST http://localhost:8130/api/notifications/send \
  -H 'Content-Type: application/json' \
  -d "{
    \"user_id\":  \"$UID\",
    \"category\": \"alerts\",
    \"title\":    \"CPU Spike\",
    \"body\":     \"prod-web-01 CPU at 94%\",
    \"url\":      \"/dashboard/alerts\"
  }" | python3 -c "import sys,json; print(json.load(sys.stdin)['notification_id'])")
echo "Notification ID: $NID"

In development without a live browser subscription, the push delivery will report no_subscription_or_vapid_not_configured. The notification record and analytics write correctly regardless — only the actual relay dispatch requires a live browser.

Step 5 — APScheduler Setup

from apscheduler.schedulers.asyncio import AsyncIOScheduler
from apscheduler.triggers.interval  import IntervalTrigger
from apscheduler.triggers.cron      import CronTrigger

scheduler = AsyncIOScheduler()
scheduler.add_job(heartbeat_check, IntervalTrigger(seconds=60),     id="heartbeat")
scheduler.add_job(daily_summary,   CronTrigger(hour=8, minute=0),   id="daily_summary")
scheduler.add_job(subscription_gc, CronTrigger(hour=3, minute=0),   id="subscription_gc")

@asynccontextmanager
async def lifespan(app):
    await init_db()
    scheduler.start()
    yield
    scheduler.shutdown()

Each scheduler job must create its own AsyncSession via async_sessionmaker. Never pass the request-scoped session to a background job — it will be closed by the time the job runs, throwing DetachedInstanceError in production under concurrent load.

Step 6 — Service Worker Push Handler

// frontend/public/sw.js

self.addEventListener('push', event => {
    const data = event.data.json();
    event.waitUntil(
        self.registration.showNotification(data.title, {
            body:    data.body,
            icon:    '/icons/icon-192x192.png',
            data:    { url: data.url, notificationId: data.id },
            actions: [
                { action: 'view',    title: 'View'    },
                { action: 'dismiss', title: 'Dismiss' }
            ]
        }).then(() => reportEvent(data.id, 'delivered'))
    );
});

self.addEventListener('notificationclick', event => {
    event.notification.close();
    event.waitUntil(clients.openWindow(event.notification.data.url));
    reportEvent(event.notification.data.notificationId, 'clicked');
});

reportEvent is a plain fetch() to POST /api/notifications/event. It fires inside event.waitUntil() so the browser keeps the service worker alive long enough for the network request to complete.

Step 7 — Frontend Subscription Hook

// Pseudo-code: usePushNotifications.js

const vapidKey   = await api.vapid.getPublicKey();
const permission = await Notification.requestPermission();
if (permission !== 'granted') return;

const sw  = await navigator.serviceWorker.ready;
const sub = await sw.pushManager.subscribe({
    userVisibleOnly:      true,
    applicationServerKey: urlBase64ToUint8Array(vapidKey)  // must be Uint8Array, not a string
});

await api.subscriptions.create({ user_id: userId, ...sub.toJSON() });

urlBase64ToUint8Array converts the URL-safe base64 VAPID public key into raw bytes. Passing the base64 string directly throws DOMException: Failed to execute 'subscribe' on 'PushManager'.

Step 8 — Preferences and Quiet Hours

# View preferences (auto-created on first subscription with role-based defaults)
curl http://localhost:8130/api/notifications/preferences/$UID

# Enable quiet hours 22:00 to 07:00
curl -X PATCH http://localhost:8130/api/notifications/preferences/$UID \
  -H 'Content-Type: application/json' \
  -d '{"quiet_hours_enabled": true, "quiet_hours_start": 22, "quiet_hours_end": 7}'

# Snooze all notifications for 60 minutes
curl -X POST http://localhost:8130/api/notifications/snooze \
  -H 'Content-Type: application/json' \
  -d "{\"user_id\": \"$UID\", \"duration_minutes\": 60}"

# Clear snooze early
curl -X DELETE http://localhost:8130/api/notifications/snooze/$UID

Step 9 — Record Events and Query Analytics

# Record a click event
curl -X POST http://localhost:8130/api/notifications/event \
  -H 'Content-Type: application/json' \
  -d "{\"notification_id\": \"$NID\", \"event_type\": \"clicked\"}"

# Pull the full analytics summary
curl http://localhost:8130/api/notifications/analytics | python3 -m json.tool

{
  "totals":    { "sent": 4, "delivered": 4, "clicked": 2, "dismissed": 2, "failed": 0 },
  "rates":     { "delivery_rate": 100.0, "ctr": 50.0, "dismiss_rate": 50.0 },
  "subscriptions": { "active": 2, "inactive": 0 },
  "by_category": [
    { "category": "alerts",       "count": 2 },
    { "category": "daily_digest", "count": 2 }
  ]
}

Build, Test and Demo

Running the Project

Without Docker

chmod +x setup.sh
./setup.sh

The script creates the full project, generates VAPID keys, installs all dependencies, runs 10 tests, starts the backend on :8130 and the frontend on :3130, then runs a live functional demo against the API. Every source file is written from heredoc — nothing needs to be downloaded separately.

With Docker

USE_DOCKER=true ./setup.sh

Starts two containers: infrawatch-backend-130 (uvicorn) and infrawatch-frontend-130 (nginx). Same ports apply.

Expected terminal output from either path:

[OK]  backend/requirements.txt
[OK]  backend/.env
[OK]  backend/generate_vapid.py
[OK]  backend/main.py
[OK]  backend/app/config.py
[OK]  backend/app/database.py
[OK]  backend/app/models/models.py
[OK]  backend/app/schemas/schemas.py
[OK]  backend/app/services/push_service.py
[OK]  backend/app/services/scheduler_service.py
[OK]  backend/app/api/subscriptions.py
[OK]  backend/app/api/notifications.py
[OK]  backend/app/api/users.py
[OK]  backend/app/api/vapid.py
[OK]  backend/tests/conftest.py
[OK]  backend/tests/test_push_notifications.py
[OK]  frontend/package.json
[OK]  frontend/vite.config.js
[OK]  frontend/index.html
[OK]  frontend/public/sw.js
[OK]  frontend/public/manifest.json
[OK]  frontend/src/main.jsx
[OK]  frontend/src/App.jsx
[OK]  frontend/src/styles.css
[OK]  frontend/src/services/api.js
[OK]  frontend/src/hooks/usePushNotifications.js
[OK]  frontend/src/pages/Dashboard.jsx
[OK]  All tests passed
[OK]  Backend running on :8130
[OK]  Frontend running on :3130

Tests

cd infrawatch-day130/backend
.venv/bin/python -m pytest tests/ -v

tests/test_push_notifications.py::test_create_user                  PASSED
tests/test_push_notifications.py::test_list_users                   PASSED
tests/test_push_notifications.py::test_create_subscription          PASSED
tests/test_push_notifications.py::test_get_user_subscriptions       PASSED
tests/test_push_notifications.py::test_get_and_update_preferences   PASSED
tests/test_push_notifications.py::test_analytics_endpoint           PASSED
tests/test_push_notifications.py::test_record_notification_event    PASSED
tests/test_push_notifications.py::test_snooze_and_clear             PASSED
tests/test_push_notifications.py::test_vapid_public_key             PASSED
tests/test_push_notifications.py::test_health                       PASSED

======================== 10 passed in ~2.7s ========================

All 10 tests use httpx.AsyncClient with ASGITransport — no background server is needed and no real push calls are made. Tests cover the complete request-response cycle including DB writes, event recording, and analytics aggregation.

Functional Verification

Work through these steps in order after ./setup.sh completes.

Health check

curl http://localhost:8130/health

{ "status": "ok", "service": "infrawatch-push-notifications", "day": 130 }

VAPID public key

curl http://localhost:8130/api/vapid/public-key

{ "public_key": "BFbDKwKfbskLDD42..." }

Create a user and subscription, send a notification, record a click

UID=$(curl -s -X POST http://localhost:8130/api/users \
  -H 'Content-Type: application/json' \
  -d '{"username":"alice","email":"alice@infrawatch.dev","role":"devops","timezone":"UTC"}' \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")

curl -s -X POST http://localhost:8130/api/subscriptions \
  -H 'Content-Type: application/json' \
  -d "{\"user_id\":\"$UID\",\"endpoint\":\"https://test.fcm.invalid/alice\",
       \"keys\":{\"p256dh\":\"BNcRd...\",\"auth\":\"tBHI...\"}}" > /dev/null

NID=$(curl -s -X POST http://localhost:8130/api/notifications/send \
  -H 'Content-Type: application/json' \
  -d "{\"user_id\":\"$UID\",\"category\":\"alerts\",
       \"title\":\"CPU Spike\",\"body\":\"prod-web-01 94%\",\"url\":\"/alerts\"}" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['notification_id'])")

curl -s -X POST http://localhost:8130/api/notifications/event \
  -H 'Content-Type: application/json' \
  -d "{\"notification_id\":\"$NID\",\"event_type\":\"clicked\"}"

curl http://localhost:8130/api/notifications/analytics | python3 -m json.tool

Confirm delivery_rate is 100.0 and ctr reflects your click event.

Swagger UI — open http://localhost:8130/docs to browse all 14 endpoints and fire test requests from the browser.

Dashboard Verification

Open

http://localhost:3130

. The dashboard has five tabs:

Tab What to confirm Overview Stat cards (Total Sent, Delivery Rate, CTR, Active Subscriptions), delivery funnel bar chart, category pie chart, recent notifications table Subscribe Browser push capability check, user selector, Subscribe / Unsubscribe button, live subscription status Preferences Four category toggles, quiet hours time pickers, snooze duration controls with active snooze display Send Test User selector, category dropdown, title/body/URL fields, four quick-fill templates Analytics CTR and delivery rate bar charts, category breakdown, metric interpretation guide

Testing real push delivery (Chrome or Firefox on localhost — no HTTPS needed):

1. Open

http://localhost:3130

1. and go to the Subscribe tab

2. Select a user and click Subscribe — the browser permission dialog appears

3. Click Allow — the subscription is saved to the backend and the status badge updates

4. Go to Send Test, select the same user, choose Alerts, fill in a message

5. Minimise the browser tab and click Send — the OS notification appears within 2 seconds

Stop All Services

./setup.sh stop

# Docker
USE_DOCKER=true ./setup.sh stop

Troubleshooting

Symptom Likely Cause Fix public_key is empty generate_vapid.py was not run cd backend && python generate_vapid.py Push service returns 403 Keys regenerated after subscriptions were saved Regenerate keys; users must re-subscribe Push service returns 410 Browser invalidated the endpoint subscription_gc handles this at 03:00 UTC; delete manually to test immediately Permission dialog does not appear Not triggered from a user gesture requestPermission() must be called inside a click handler sw.js returns 404 Service worker not at root scope File must be in frontend/public/, not frontend/src/ DetachedInstanceError in scheduler Job sharing the request-scoped session Create a fresh AsyncSession inside every scheduler job DOMException on subscribe() VAPID key passed as string, not bytes Run urlBase64ToUint8Array() before passing to applicationServerKey

YouTube Demo Link:-

Today you will wire up a production-grade offline-capable task management system. By the end of this lesson your application will:

• Store data locally in the browser using IndexedDB

• Detect network state and queue writes when offline

• Sync queued changes to the server the moment connectivity returns

• Detect and resolve data conflicts when server and client diverge

• Run background sync via a Service Worker — no user action required

This slots directly into the Week 11 infrastructure around team collaboration. Users on unstable connections — field technicians, remote workers, mobile users — can keep working without hitting a 503. Their edits land on the server when the network recovers.

Before You Start

Prerequisites:

• Python 3.11+, Node 20+, Docker (optional)

• Day 128 backend available as reference

• Browser: Chrome or Edge — they have full Background Sync API support. Firefox handles reconnect sync through the foreground online listener instead.

The offline pipeline has four stages, in order:

User Action
  → IndexedDB write    (immediate — user never waits)
  → Sync Queue append
  → pushSync() drain   (on reconnect / 30s interval / SW event)
  → FastAPI /api/sync/ (conflict check, merge, return server delta)

Every stage is decoupled. The user never waits for the network. Keep this mental model in your head as you build — every decision in this lesson traces back to it.

Why Offline-First Matters in Production

Google Docs autosaves every keystroke locally before it ever reaches a Google server. Notion queues every block edit in IndexedDB. Figma stores the entire file graph client-side and syncs deltas. These are not edge-case features — they are table-stakes for any collaborative tool.

The discipline is called offline-first: writes happen locally first, sync happens second. This inverts the classic “optimistic update” pattern (assume the network works) into something more resilient — assume the network is gone, sync opportunistically.

The Core Architecture

Read more

InfraWatch AsyncOps — a production background-processing console that ships real work off the request path:

1. Celery execution layer — Multi-queue routing, retries, worker scaling, Flower visibility

2. Async task catalog — Metrics collection, batch processing, notifications, reports, maintenance

3. Cron scheduler — User-defined jobs with priority and manual execution

4. Workflow orchestration — DAG dependencies, conditional nodes, chord/chain patterns

5. Monitoring plane — WebSocket streaming, Prometheus metrics, queue depth alerts

6. Operator console — Datadog/Celery-style dark dashboard with live controls

Core Concept: Request Path vs Worker Path

User-facing APIs must return in milliseconds. Anything that touches disks, networks, or aggregation belongs on a worker path — queued, retried, and observed separately. Stripe and GitHub run variations of this split: HTTP acknowledges intent; workers guarantee completion.

The non-obvious detail: queue naming is a capacity contract, not decoration. A metrics.high queue with dedicated workers prevents report generation from starving health checks — the same pattern AWS SQS priority queues and Sidekiq weighted queues use.

Where This Fits

Metrics Collection          telemetry source
        │
Background Processing       ◀ this lesson
        │
Alert Management            evaluates completed work & thresholds
        │
Notifications               delivers outcomes

AsyncOps is the execution backbone between raw metrics and human response. Alerts assume tasks complete reliably and emit measurable latency.

Component Architecture

API — FastAPI enqueues tasks, exposes stats, and streams WebSocket updates.
Broker — Redis holds queues and result backends with isolated DB indexes.
Workers — Celery consumers per queue family with autoretry and progress metadata.
Scheduler — Croniter loop plus Celery Beat for platform housekeeping.
Orchestrator — DAG engine resolves dependencies before dispatching child tasks.
Console — React operator UI with queue pause/resume, retry/cancel, workflow triggers.

Read more

→ Subscribe now to Access All paid lessons & private source code repository - 250 + coding lessons.

Are you preparing for Cracking FAANG interviews ? Here is full Preparation kit: 52 FAANG System Design Interview Questions

→Download the free Interview Pack

What We’re Building Today

By the end of this lesson you will have a fully mobile-optimized InfraWatch dashboard that feels and behaves like a native app. Here’s the agenda:

• A bottom navigation bar with badge counts — the pattern used by PagerDuty, Datadog mobile, and New Relic

• Swipe-to-dismiss gesture on alert items — no buttons, just touch

• Pull-to-refresh with a distance threshold and haptic feedback

• Safe area insets so the UI doesn’t get swallowed by iPhone notches and Android chin bars

• 44px minimum touch targets throughout — the Apple HIG and Google Material Design minimum

• Real system metrics from psutil served by FastAPI, updating every 30 seconds

This builds directly on Day 127 (PWA), taking the installable shell and making it genuinely usable with one thumb.

Why Mobile Optimization Is a Serious Engineering Problem

Every DevOps engineer has been there: a 3 a.m. page on your phone, you open the dashboard, and the table is a pixel-wide soup of compressed columns you can’t read, let alone tap. Grafana, Datadog, and PagerDuty all shipped dedicated mobile apps or heavily invested in mobile web precisely because an unoptimized monitoring dashboard under incident pressure is actively dangerous — you mis-read a number, you tap the wrong button.

Mobile optimization is not just “make it responsive.” It covers three separate problems:

1. Input model mismatch — Touch has no hover state, no right-click, 50ms tap delay (unless you remove it), and a 300ms ghost-click issue browsers introduced in 2014 that still bites developers who don’t set touch-action: manipulation.

2. Layout model — 100vh breaks on iOS Safari when the address bar shows/hides. The correct unit is 100dvh (dynamic viewport height), available since Safari 15.4.

3. Interaction affordance — A user cannot swipe something they don’t know is swipeable. Design for discoverability: a slight peek of the red dismiss background before the user even swipes tells them what will happen.

Read more

By the end of today, InfraWatch will install on a phone or desktop like a native app, work during a network outage using cached data, receive push alerts from our FastAPI backend, and silently queue failed API calls for retry when connectivity returns. Here is what we cover:

• PWA manifest and installability via vite-plugin-pwa

• A hand-written service worker with four caching strategies

• VAPID key generation and push notification dispatch from FastAPI

• Background sync queue for offline action replay

• The App Shell architecture that makes InfraWatch open in under 200 ms

Part One: Concepts

The Three Pillars

A Progressive Web App is any web app that ships three things together: a Web App Manifest (JSON file declaring the app identity), a Service Worker (a background JavaScript thread with network interception power), and a secure context (HTTPS or localhost). That is the complete definition. The rest — push notifications, background sync, install prompts — layers on top.

Twitter Lite, Pinterest, and Starbucks were not built by rewriting their apps. They added these three things and unlocked 30–70% improvements in engagement metrics.

Service Worker: A Programmable Network Proxy

The most accurate mental model for a service worker is a programmable reverse proxy embedded inside the browser. Every fetch() call from your React app passes through it. Once active, the SW controls all network traffic from its scope, across all open tabs matching that scope.

The lifecycle has four states to internalize:

State What Happens Installing SW downloads; install event fires; you pre-cache the app shell Waiting Old SW still controls pages; new SW waits for all tabs to close Activating New SW takes over; activate fires; old caches are pruned Activated SW intercepts every fetch; caching logic runs here

The “waiting” state is where most PWA bugs appear. A new SW silently sits installed but inactive until all pages using the old SW close. Calling self.skipWaiting() in install bypasses this — useful in development, handled carefully in production.

Four Caching Strategies — Choosing a Policy

Every caching strategy is a tradeoff between freshness and availability. Pick based on how often your data changes.

Cache-First — For static assets (JS bundles, fonts, icons). Check cache first; if missing, fetch from network and cache it. The InfraWatch app shell uses this — the skeleton renders in under 100 ms even on a slow connection.

Network-First — For API data. Always attempt a live request. On failure, serve the cached response. InfraWatch uses this for metric data — you always see the most recent numbers available.

Stale-While-Revalidate — Serve cache immediately, update in background. Twitter’s timeline uses this — you see last-known content instantly while new data arrives silently.

Cache-Only — Serve from cache only, never touch the network. Used for the offline fallback page.

Read more

🗄️ 52 FAANG Interview Question Vault — $299 one-time

All 52 walkthroughs + 52 drill cards + 6 framework cheatsheets + the Senior+ System Design Playbook. Everything, immediately. Lifetime access with all future updates included.

→ [VAULT CHECKOUT LINK]

What We Are Building

This lesson delivers a production-grade metrics collection and observability console for InfraWatch:

1. Time-series schema — Raw points, retention policies, metric catalog, and pre-aggregated rollups

2. Collection pipeline — Agent protocol, validation, batch ingestion, and scheduled collection

3. Storage layer — PostgreSQL persistence with Redis hot-path cache and pub/sub streaming

4. Aggregation engine — Sliding-window statistics, historical rollups, and trend detection

5. Query API — Time-range queries with caching, CSV/JSON export, and summary endpoints

6. Observability console — Datadog-style dashboard with live charts and agent status

Core Concept: Write Path vs Read Path

Production metrics systems split into two lanes. The write path accepts high-volume, fire-and-forget data points from agents—validate, index, persist, and push to a hot cache in milliseconds. The read path serves dashboards and alerts—query pre-aggregated buckets first, fall back to raw data only when needed, and cache results by time range.

The insight most tutorials skip: retention is a write-time decision, not a cleanup cron job. When a point lands, the system already knows its tier (realtime, recent, daily) and expiry date based on metric name patterns. Datadog and Grafana Cloud use this pattern—policy resolution at ingest keeps query performance predictable as volume grows.

Where This Fits in the Overall System

Server Fleet Management        inventory spine — agents attach here
        │
Metrics Collection             ◀ this lesson
        │
Alert Management               rules evaluate against stored series
        │
Notification System            delivers when thresholds breach

You are building the telemetry pipeline that turns raw host signals into queryable time-series. Every alert rule and capacity plan downstream assumes this layer exists and returns data within SLA.

Component Architecture

Agents — Lightweight collectors (psutil-based) push validated batches over WebSocket every five seconds.

Ingestion — Validator rejects out-of-range values; ingester dual-writes to PostgreSQL and Redis; catalog index updates on every point.

Aggregation — Background collector plus sliding-window engine compute avg/min/max/p95; rollups compress raw data into 1m/5m/1h buckets.

Query API — FastAPI serves cached queries, exports, summaries, and SSE streams for live dashboards.

Console — Dark-theme React UI polls REST every five seconds and receives WebSocket push on new points.

Read more

• A live FastAPI server with 18 documented endpoints — auth, users, teams, metrics, security

• A Python SDK (InfraStackClient) with chainable resource namespaces, typed errors, and exponential backoff

• 45 integration tests across 5 suites — all passing before we move on

• A React dashboard styled after Postman’s collection runner — live test execution, latency charts, status distribution

• A single self-contained setup.sh that builds, tests, and demos the whole thing in one command

Prerequisites: Python 3.11+, Node.js 20+, Docker (optional) Completed: Days 122–125 (API Security, Documentation, Testing, SDKs)

Project Structure

Everything in this project is created by setup.sh — you don’t manually create anything.

day126/
├── setup.sh                   # one-command master script
├── docker-compose.yml
├── backend/
│   ├── main.py                # FastAPI app (18 endpoints)
│   ├── requirements.txt
│   ├── sdk/
│   │   └── client.py          # Python SDK (InfraStackClient)
│   └── tests/
│       ├── conftest.py
│       └── test_integration.py  # 45 tests, 5 suites
├── frontend/
│   ├── package.json
│   └── src/App.jsx            # React dashboard
└── diagrams/

Why Integration Testing Deserves Its Own Day

You can have 100% unit test coverage and still ship a broken API. Unit tests verify functions in isolation; integration tests verify the system as a whole — authentication gates holding, database writes surviving network roundtrips, SDK error handling matching actual server behavior.

Stripe’s payment API has a dedicated integration sandbox because even minor contract drift between client and server can cause millions in failed transactions. GitHub’s API runs end-to-end tests on every deployment. This isn’t paranoia — it’s engineering maturity.

Today we prove our Week 11 API stack actually works, together. Every piece we built across Days 120–125 gets validated end-to-end: auth, users, teams, metrics, security headers, SDK contracts, documentation coverage, and latency baselines.

The Five Test Dimensions

Every production API needs validation along five distinct axes. Miss any one and you ship blind. Our 45-test suite covers all five — a standard you’ll carry into every job you take.

Read more

🗄️ 52 FAANG Interview Question Vault — $299 one-time

All 52 walkthroughs + 52 drill cards + 6 framework cheatsheets + the Senior+ System Design Playbook. Everything, immediately. Lifetime access with all future updates included.

→ [VAULT CHECKOUT LINK]

What We Are Building

This lesson delivers a production-grade server fleet control plane for InfraWatch:

1. Inventory domain — Rich server models with lifecycle state, health telemetry, tags, dependencies, and audit history

2. Operations API — CRUD with pagination, soft-delete, advanced search, bulk actions, groups, and templates

3. Validation layer — IP/hostname checks, TCP connectivity, HTTP health, SSL expiry, and network discovery

4. Credential plane — SSH key generation, encrypted storage, rotation, and connection testing

5. Live console — Datadog-style React dashboard with real-time WebSocket health updates

Core Concept: Inventory as the Source of Truth

Every metrics pipeline, alert rule, and automation workflow in a platform like Datadog or PagerDuty starts with one question: which servers exist right now, and are they reachable? Server inventory is not a spreadsheet—it is the authoritative graph that downstream systems query before they act.

The insight most tutorials skip: lifecycle state and health state are different axes. A server can be active in provisioning terms but degraded in health terms. Mixing them into one status field breaks autoscaling logic (which cares about lifecycle) and paging logic (which cares about health). InfraWatch Fleet keeps both dimensions explicit.

Where This Fits in the Overall System

Authentication Foundation     identity gate for all API calls
        │
Server Fleet Management       ◀ this lesson
        │
Metrics Collection            time-series from known inventory
        │
Alerts & Notifications        rules evaluated per server identity

You are building the inventory spine that metrics agents attach to and alert engines evaluate against. Without it, alerts fire on hostnames nobody registered—or worse, on hosts that were decommissioned last week.

Component Architecture

Presentation — Dark-theme React console: fleet overview metrics, searchable inventory grid, validation tools, credential manager. Polls REST every 15 seconds; WebSocket pushes health deltas instantly.

API — Flask blueprints under /api/v1. Thin routes delegate to services: server_service, health_service, ssh_service, bulk_service.

State — PostgreSQL stores servers, tags, groups, templates, audit logs, SSH keys. Redis caches health snapshots and increments check counters.

Runtime — Background health monitor runs every 30 seconds. Docker Compose orchestrates Postgres 16, Redis 7, API, and UI.

Read more

By the end of this lesson you will understand:

• What an SDK actually is — and why it is architecturally distinct from a wrapper

• The three-layer model that every production SDK follows (transport, resources, exceptions)

• How token lifecycle management works and why it must live inside the SDK

• The typed exception hierarchy pattern and why it beats raw HTTP status codes

• How this SDK layer connects everything we built this week into a usable product surface

What an SDK Actually Is

Most engineers describe an SDK as “a wrapper around an API.” That description is accurate but misleading. A wrapper hides HTTP. An SDK owns the client-side failure surface.

The distinction matters enormously at scale. When Stripe ships a Python SDK, they are not saving developers the ten minutes it takes to write requests.post(). They are ensuring that every authentication failure, rate limit event, network timeout, and validation error is caught, classified, and surfaced as a typed exception — consistently, across every company that integrates Stripe, forever.

Without an SDK, each consumer writes their own error handling. Some check .status_code. Others parse .json()["error"]["message"]. A few catch requests.exceptions.Timeout. The result is thirty different error-handling strategies in thirty codebases, each fragile in its own way. When you push a breaking change to your API response shape, all thirty break in different places.

With an SDK, there is one place where that contract lives. One patch fixes everyone.

This is the calculation Twilio made in 2008, Stripe made in 2011, and GitHub made with Octokit in 2015. For InfraManager, Day 125 is that same decision point. The Week 11 API is solid. Now we make it trustworthy to integrate.

The Three-Layer SDK Model

Every production SDK — regardless of language, company, or domain — follows the same three-layer architecture. Understanding these layers is more valuable than understanding any specific implementation.

Layer 1 — The Transport

The transport layer is the engine room. Its job is to convert method calls into HTTP requests and HTTP responses into either data or exceptions. Consumers never interact with it directly.

The transport handles four responsibilities: building request headers (authentication, content type, API version), enforcing timeouts, managing the connection pool, and handing off error responses to the exception layer.

In an async SDK (which InfraManager uses), the transport client is async-native from the start. Wrapping a synchronous HTTP client in async code causes subtle deadlocks under load — a mistake that plagues SDKs built quickly and patched slowly. Our Python transport uses httpx.AsyncClient natively. The JavaScript transport uses fetch with AbortController for timeout enforcement, since the browser’s native fetch has no built-in timeout.

One detail worth internalising: the transport does not retry. That is a separate concern. The transport makes one clean attempt, captures the result or the exception, and surfaces it. Retry logic lives in the consumer’s workflow, where it belongs — because the consumer knows whether a given operation is idempotent.

Layer 2 — The Resources

Resources are the public surface of the SDK. They are intentionally thin. A UsersResource exposes list, get, create, update, and delete. Each method maps to exactly one REST verb and path. No business logic lives here.

The resource layer’s only responsibility is to accept typed parameters, construct the correct path and payload, call the transport, and deserialize the response into a typed object.

This thinness is architectural discipline, not laziness. When Stripe’s Python library adds a stripe.customers.list() method, there is almost no decision-making in that method. The complexity lives in the transport and the data models. Resources stay readable because they are designed to read like API documentation.

In InfraManager, the three resources — AuthResource, UsersResource, TeamsResource — follow this pattern exactly. Each hangs off the central InfraClient object, which is the only import a consumer needs.

Layer 3 — The Exceptions

This is where most SDKs are mediocre and a handful are excellent. The difference is almost entirely about whether exceptions carry typed, actionable information or generic HTTP noise.

Compare these two error-handling experiences:

Mediocre SDK:

HTTPError: 409 Conflict

Excellent SDK:

ConflictError: Email already registered
  .status_code = 409
  .message = "Email already registered"
  .request_id = "req_7f3a2c"

The mediocre SDK forces the consumer to parse error messages as strings. The excellent SDK gives the consumer a named class they can catch precisely, with structured fields they can log, display, or act on.

InfraManager’s exception hierarchy maps every meaningful HTTP status code to a named, catchable exception class. All inherit from a single InfraSDKError base, so consumers can catch broadly or narrowly depending on their context:

HTTP Status Exception Class Actionable Signal 401 AuthenticationError Re-authenticate 403 PermissionError Check RBAC role 404 NotFoundError Handle missing resource 409 ConflictError Duplicate resource 422 ValidationError Check .errors list 429 RateLimitError Wait .retry_after seconds 5xx ServerError Retry with backoff Network NetworkError Check connectivity

The RateLimitError exception is worth a moment. On Day 121 we built the server-side rate limiting system. Today, the SDK reads the Retry-After header from the 429 response and populates exception.retry_after — so the consumer has everything they need without parsing HTTP headers manually.

Token Lifecycle: Why It Lives Inside the SDK

Token management is the most commonly misimplemented part of API integration. Most teams either pass tokens in every call (verbose), refresh them too aggressively (wasteful), or refresh them too lazily (broken sessions).

The correct pattern is a TokenStore inside the transport that handles this invisibly:

When client.auth.login() succeeds, the JWT is stored internally with a TTL set to the token’s expiry minus a 60-second buffer. Every subsequent request checks whether the stored token is still valid before injecting the Authorization header. If it has expired, the store returns nothing — and the transport omits the header, causing the server to return 401, which surfaces to the consumer as AuthenticationError. They then call client.auth.login() again and the cycle continues.

The consumer never touches a token string after the initial login. This is the experience GitHub’s Octokit delivers, and it is one of the primary reasons Octokit adoption overtook raw fetch calls within GitHub’s own developer community.

The auth state machine has five named states: UNAUTHENTICATED, AUTHENTICATING, AUTHENTICATED, TOKEN_EXPIRED, and LOGGED_OUT. The transitions between them are deterministic. When you call client.auth.logout(), the store is cleared and the client returns to UNAUTHENTICATED. When TTL elapses, the next request attempt discovers TOKEN_EXPIRED and surfaces an AuthenticationError. No silent failures. No stale tokens silently accepted.

Python vs JavaScript SDK: Same Contract, Different Runtimes

InfraManager ships SDKs in both Python and JavaScript with intentionally identical public surface areas. A Python developer and a JavaScript developer looking at the SDK documentation see the same resources, the same exception names, the same method signatures.

The internal differences are runtime-specific:

Python SDK — uses httpx.AsyncClient with Python’s native asyncio. The entire SDK is async-native with async/await. Consumers use it inside async def functions or asyncio.run().

JavaScript SDK — uses the fetch API with AbortController for timeout enforcement. Ships as ESM (ECMAScript Modules) so it works in Node 18+, modern browsers, and Cloudflare Workers without bundler configuration.

Both SDKs use TokenStore for credential management. Both define the same exception hierarchy by name. Both auto-inject the Authorization header. The runtime differences are fully encapsulated — the consumer experience is identical.

This is the same commitment GitHub makes with Octokit — @octokit/rest for JavaScript and octokit.py for Python share the same design language — and it is a significant adoption driver because teams that use both languages only need to learn the SDK once.

SDK Testing: Why ASGITransport Changes Everything

SDK testing has a well-known trap: if you mock the HTTP layer, you are testing the mock, not the SDK. If you spin up a real server for tests, you add infrastructure dependencies and flakiness.

The solution for FastAPI-backed SDKs is httpx.AsyncClient with ASGITransport. This routes the SDK’s HTTP calls through the FastAPI app in-process, with no sockets, no ports, and no process management. The SDK executes identically to how it would against a live server — same middleware, same ORM, same exception propagation — but the tests are hermetic and fast.

This matters architecturally: the 36 tests on Day 125 found two real bugs that mocks would have missed:

1. A duplicate route registration in the auth router where a placeholder GET /me handler returned None, shadowing the real authenticated handler. In production this would have caused every /me call to return a 500.

2. SQLAlchemy lazy-loading a relationship synchronously inside an async ASGI context. len(team.members) looks harmless in code review but triggers a synchronous DB call that raises MissingGreenlet under ASGI transport. The fix — an explicit SELECT COUNT(*) against the association table — is architecturally correct for async ORMs and would never have been caught by HTTP mocks.

The test suite is layered into three categories that mirror production concerns:

Unit tests validate the exception hierarchy in isolation — no database, no HTTP, no FastAPI. Nine tests verify that raise_for_status(401, ...) raises AuthenticationError, that RateLimitError.retry_after populates correctly, and that the inheritance chain is intact. These run in milliseconds.

Integration tests validate every resource method against the live app — register, login, create user, update role, add team member, delete. Twenty-two tests cover the full happy path and all error paths.

Contract tests validate response shape — that User.id is a string, User.is_active is a boolean, UserList.total is an integer. Five tests. These guard against silent schema drift that would break consumers without failing unit or integration tests.

The InfraManager SDK Playground Dashboard

GitHub Link:-

https://github.com/sysdr/infrawatch-fullstack-p/tree/main/day125/project

The React dashboard at port 3000 is not a demo — it is a functional internal tool built on top of the same JavaScript SDK that ships to consumers. It demonstrates four capabilities:

Users tab — live paginated user list, role-filtered views, create form with immediate feedback, delete with optimistic UI. All mutations go through client.users.create() and client.users.delete() — the same SDK calls a consumer would write.

Teams tab — team cards with live member counts, add-member dropdown that cross-references the user list, instant count updates after member operations.

SDK Explorer — syntax-highlighted, copy-ready code examples in both Python and JavaScript for every common workflow: login, list users, create team, add member, error handling.

API Docs tab — the full endpoint reference table generated from our Week 11 API surface, showing method, path, and description for all 15 endpoints.

The dashboard’s dark green aesthetic is intentional — it mirrors the design language of professional infrastructure tooling (Datadog, Grafana, AWS console) rather than generic admin UIs. Engineers who use professional tools respond better to dashboards that look like professional tools.

Key Design Decisions Worth Remembering

Single entry point. InfraClient is the only thing a consumer imports. All resources hang off it. This mirrors Stripe (stripe.customers.create()), GitHub (octokit.rest.users.list()), and Twilio (client.messages.create()). A single entry point means one object to configure, one context manager to use, one thing to mock in tests.

Async context manager. async with InfraClient("http://...") as client: ensures the underlying connection pool is properly closed. This is not a nice-to-have — unclosed httpx.AsyncClient instances in production cause connection pool exhaustion over time.

No retry logic in the SDK. The SDK classifies errors and surfaces them cleanly. Retry decisions belong to the consumer because they know the idempotency characteristics of their workflow. An SDK that retries opaquely creates silent duplicate operations in non-idempotent contexts.

Typed dataclasses, not raw dicts. Every response is deserialized into a Python dataclass or JavaScript object with named properties. user.email is safer than response["email"] — it fails loudly at the point of deserialization rather than silently returning undefined or raising KeyError somewhere downstream.

Assignment

Extend the Python SDK with a SearchResource that wraps a GET /api/v1/users/search?q=<query> endpoint.

The endpoint should filter users by full_name or email using a case-insensitive SQL LIKE query. The SDK resource should follow the same pattern as UsersResource — a single search(query: str) -> list[User] method. Wire it into InfraClient as client.search.

Write three tests:

1. A search that matches at least one user returns a non-empty list of typed User objects

2. A search with no matches returns an empty list without raising an exception

3. An unauthenticated search raises AuthenticationError

Hints:

• Backend filter: User.full_name.ilike(f"%{q}%") | User.email.ilike(f"%{q}%")

• Return the same {"items": [...], "total": N} envelope as the users list endpoint

• The SearchResource is six lines of code if you pattern-match against UsersResource.list()

Success Criteria

You have completed Day 125 when:

• pytest sdk/python/tests/test_sdk.py reports 36 passed, 0 failed

• The React dashboard loads at

http://localhost:3000

• , you can log in and interact with Users and Teams

• You can copy the Python SDK example from the SDK Explorer tab, paste it into a Python shell, run it against the live backend, and see real data returned as typed objects

• You understand why TokenStore must live inside the transport and not be managed by the consumer

• You can explain the difference between a transport error, a resource method call, and an exception class — and which layer owns each

YouTube Demo Link:-

By the end of this lesson you will have a working, automated API test suite for the User & Team Management backend we built across Week 11. The suite covers five distinct testing layers — unit, integration, contract, load, and security — all wired together under a single setup.sh script that outputs results to a live React dashboard. Nothing is mocked. Every test hits the real FastAPI server.

Today’s agenda:

• Understand the API testing pyramid and why each layer exists

• Write unit and integration tests with pytest and httpx

• Implement JSON Schema contract testing

• Run Locust load tests and capture p95 latency

• Run bandit security scans and custom OWASP surface checks

• Combine all results and stream them into a React test dashboard

Read more

By the end of this lesson you will have a fully operational API documentation portal — the kind that makes engineers trust your API before they write a single line of code against it. Here is what gets built:

• A versioned OpenAPI 3.1 specification generated programmatically for both v1 (deprecated) and v2 (current) API surfaces, complete with security schemes, contact info, and server blocks.

• A code sample engine that generates Python, JavaScript, and cURL snippets for every endpoint automatically — no manual copy-paste.

• A deprecation lifecycle system that brands the v1 spec with deprecated: true, x-sunset-date, Deprecation and Sunset response headers, and a Link: successor-version header on every response.

• An InfraDocs React portal — a dark-mode API explorer that mirrors the interaction patterns of Stripe Docs and ReadMe.io, with a Try It panel, SDK snippet generator, and version comparison view.

Prerequisites

• Day 122 complete — API key management and auth patterns are in place

• Python 3.11+, Node.js 20+, Docker (optional but recommended)

• Comfortable with FastAPI routing and Pydantic models

Read more