Skip to main content

From Measurements to a Release Decision

This lesson extends probability into experimental judgment. Prerequisites: conditioning, covariance, confidence language, and the workload worked example. Allow 2–3 hours including a one-page decision memo.

State the decision and estimand​

A release manager asks whether version B makes requests faster. Translate that into a quantity: the change in mean latency for a specified target workload, the change in probability of exceeding200 ms, or a particular quantile. These are different questions. State a practically meaningful improvement or regression threshold before collecting the comparison.

The fixture in Lab D supports descriptive group means and standardization. It does not contain the evidence needed for a causal release decision. A defensible memo can conclude that more measurement is needed while still explaining exactly what the existing data shows.

Choose units and assignment​

If requests within a session share state, randomizing individual requests can cause interference or carryover between versions. One possible design assigns entire sessions to A or B; another uses isolated benchmark processes. Choose based on the system, state the unit, and analyze dependence at that level. More requests from the same session are not automatically more independent experimental units.

Record version, randomization unit, workload group, environment, time, warm-up policy, sample-selection rule, failures, and latency. Define treatment of timeouts and dropped requests in advance; excluding them can make a failing version look fast. Preserve comparable resource limits and intended cache conditions.

Separate random error from systematic error​

Random assignment addresses some confounding on average, but it does not repair instrumentation that records one version differently or outcomes missing because a service crashed. Check collection coverage and balance using a planned procedure. Stratify or standardize using the target workload specified before viewing the result, not weights selected to favor one version.

For independent sample means, the variance of their difference is the sum of their variances. For a paired design it includes covariance, and the natural observations are paired differences. Means and counts alone do not provide these variances. An interval about a mean says nothing directly about p99.

Plan stopping and interpretation​

For this module, fix the sample size or collection window and primary metric before the experiment. Repeatedly checking a fixed-sample interval until a desired result appears changes the procedure. Production sequential decisions require a method designed for repeated looks. Exploring twenty metrics and reporting only the most favorable one creates a related selection problem.

Interpret the effect size and interval against the practical threshold. "Did not detect a difference" is not evidence that two versions are equivalent. "Statistically detectable" does not by itself make a change operationally valuable.

Memo task and review criteria​

Using Lab D's fixture, write a one-page memo with:

  1. Raw and 50/50 standardized means, showing the arithmetic and target population.
  2. A concrete reason why workload composition changed the aggregate conclusion.
  3. Two plausible alternative explanations besides the version causing latency changes.
  4. A proposed collection and assignment plan, sampling unit, primary metric, stopping rule, and practical decision threshold.
  5. A statement of what remains unknown and what result would change the release decision.

Pass: all five items are present, the calculations are correct, and the decision is limited to what the evidence supports. Revise: any causal claim based solely on the fixture, averaging of percentiles, fabricated variability, or treatment of copied rows as independent observations.

For a changed case, use an 80/20 target mix. The standardized means become A=28 ms and B=31.6 ms. This agreement in direction still does not establish causality. Carry this discipline into the later benchmarking, observability, and experiment-design work.

Continuous models and inference extension​

Append the inference decision memo, preserving the distinction between the synthetic durations, Bernoulli model and paired fixture. The original five criteria still apply; the appendix adds model-specific evidence.