Skip to main content

Smoke Tests After Each Deploy

What This Concept Is​

A smoke test is a small, real check that runs against a real environment after a deploy, not before a merge. It answers one question: "did the thing we just deployed actually start serving traffic correctly?"

Three checks are enough for a capstone:

  • Liveness: the process answers an HTTP request on its health endpoint with 200. Confirms the container started, ports are bound, the runtime is alive.
  • Readiness with a real dependency: an endpoint that exercises the database (and other critical dependencies) returns expected shape -- not just 200, but a JSON body with "db":"ok". Confirms wiring.
  • Critical user path: one authenticated request that represents a core feature (e.g., POST /login + GET /me). Confirms end-to-end business behavior against real data.

These are not unit tests. They run in production. They are cheap, broad, and wrong-signal-cheap-to-investigate. Google SRE calls this category "production probes" and treats it as a first-class reliability tool, distinct from unit/integration testing.

Why It Matters Here (In the Capstone)​

Smoke tests are the evidence that a deploy succeeded in reality, not just in CI. They are what turns "my pipeline went green" into "users can use the system." Every subsequent piece of this module -- rollback triggers, release notes, runbook -- depends on smoke tests existing.

Without post-deploy smoke, a deploy is a hope. Env vars could be missing, the new binary could fail to connect to the DB on a specific TLS flag, the rewrite rule for a new endpoint could be wrong, the DNS might still point at the blue revision. Unit and integration tests in CI cannot catch any of those; only a smoke against the real environment can.

Concrete Example(s)​

A smoke script that runs as the last pipeline step:

#!/usr/bin/env bash
set -euo pipefail
BASE_URL="${1:-https://api.capstone.example.com}"
TIMEOUT=10

# 1. Liveness
curl -fsSL --max-time "$TIMEOUT" "$BASE_URL/healthz" > /dev/null
echo "[ok] liveness"

# 2. Readiness with DB
READY=$(curl -fsSL --max-time "$TIMEOUT" "$BASE_URL/readyz")
echo "$READY" | grep -q '"db":"ok"' || { echo "[fail] db not ok: $READY"; exit 2; }
echo "[ok] readiness with db"

# 3. Critical path: login smoke user, fetch /me
TOKEN=$(curl -fsSL --max-time "$TIMEOUT" -X POST "$BASE_URL/login" \
  -H "Content-Type: application/json" \
  -d '{"email":"smoke@capstone.test","password":"'"${SMOKE_PW}"'"}' \
  | jq -r .token)
[[ -n "$TOKEN" && "$TOKEN" != "null" ]] || { echo "[fail] login returned no token"; exit 3; }

ME=$(curl -fsSL --max-time "$TIMEOUT" "$BASE_URL/me" -H "Authorization: Bearer $TOKEN")
echo "$ME" | grep -q '"email":"smoke@capstone.test"' || { echo "[fail] /me wrong shape"; exit 4; }
echo "[ok] critical path"

echo "SMOKE PASS"

In the workflow:

- name: Smoke test
  env:
    SMOKE_PW: ${{ secrets.SMOKE_PW }}
  run: ./scripts/smoke.sh https://api.capstone.example.com
  timeout-minutes: 3

If any check fails, the deploy job fails, which triggers the rollback path from concept 10. For a canary or blue/green deploy, the smoke also gates traffic cut-over:

./scripts/smoke.sh "$CANARY_URL" && gcloud run services update-traffic capstone-api --to-latest

A readiness endpoint that actually reports its truth:

app.get("/readyz", async (_req, res) => {
  const db = await checkDb().catch((e) => e.message);
  const cache = await checkCache().catch(() => "unavailable");
  const ok = db === "ok";
  res.status(ok ? 200 : 503).json({ db, cache });
});

Common Confusion / Misconceptions​

  • "Smoke tests are a subset of integration tests." No. Integration tests run in CI against a local or staging service, with fixtures. Smoke tests run against prod (or the environment you just deployed) with real secrets and real infrastructure. The purpose is different: integration tests catch bugs in code; smoke tests catch bugs in deployment.
  • "/healthz returning 200 is enough." /healthz often only checks that the process is alive. It does not check that the new code is wired correctly to the database, the secret store, or the external dependencies. You need at least one check that exercises the real data path.
  • "Smoke tests should run against every endpoint." No -- that is load testing with extra steps. Smoke covers the 2-3 paths whose failure would be catastrophic; everything else is caught by monitoring and alerts within minutes, not seconds.
  • "A smoke failure is a rollback." Usually, but not always. Distinguish between "the deploy failed" (rollback) and "the smoke user's password expired" (fix smoke, re-run). The smoke script's exit code should be trustable; a flaky smoke erodes the entire discipline.
  • "Smoke can live outside the pipeline." It can, but it should also run inside the pipeline as the final deploy step. External probes catch ongoing outages; in-pipeline smoke catches bad deploys before traffic shifts.

How To Use It (In Your Capstone)​

  1. Write /healthz (liveness) and /readyz (readiness) endpoints in the app; /readyz should actually talk to the DB and secret store.
  2. Create a dedicated smoke user (deactivate-able) per environment, with a password stored only in the secret store.
  3. Commit the smoke script under scripts/smoke.sh. Run it as the last deploy step, with a hard timeout.
  4. Keep total smoke time under 30 seconds. Anything slower belongs in post-deploy monitoring, not the deploy gate.
  5. Review the smoke script any time a new major endpoint ships -- the critical path may have moved; the old smoke is now incomplete.
  6. For canary/blue-green deploys, run smoke against the new revision before shifting traffic.
  7. Log the smoke result in the release note (concept 14) so the change log answers "was this deploy verified?"

Smoke-as-Continuous-Probe​

The same smoke script can double as a continuous probe run every 1-5 minutes from an external location (Cloud Monitoring uptime check, AWS Route53 health check, a cheap cron on an outside box). In-pipeline smoke catches bad deploys at deploy time; continuous smoke catches drift or external-dependency failures between deploys. Use both, and make sure they share the exact same smoke user and critical path so findings are comparable.

See also (integrative)​

Check Yourself​

  1. Which three checks does your smoke test run, and which one has failed most recently?
  2. Which smoke failure would trigger an automatic rollback? Which would not (false positive)?
  3. Where does the smoke user's password live, and who can rotate it?
  4. What is the smoke's hard timeout, and when was it last exceeded?
  5. Does /readyz actually hit the DB, or does it just return 200? How did you verify?
  6. For a canary, does smoke run against the canary URL before traffic shift, or after?

Mini Drill or Application (Capstone-scoped)​

  1. Three-check smoke (45 min). Add /healthz and /readyz to the app, create a smoke user, commit scripts/smoke.sh, wire it as the last deploy step.
  2. Red-on-purpose. Deliberately break one endpoint (misspell a DB column in /readyz) in a branch, push, and watch the pipeline fail at the smoke step. Confirm the failure exits with a nonzero code and prints a legible message.
  3. Smoke audit. Review the smoke after a new endpoint ships. Does the critical path still reflect a real user flow? If not, update the smoke and note the change in CHANGELOG.md.

Source Backbone​

Capstone deployment applies cloud, delivery, and operations material. These books are the source backbone for the delivery decisions.