Skip to main content

Case Studies

These are synthetic engineering scenarios, not reported production incidents. They extend the required practice with concrete data and reviewable decisions. Use the book and topic guide for underlying probability; calculations below are original worked exercises.

1. A conversion experiment with limited evidence​

Version A converts 10 of 100 independent visitors; B converts 12 of 100. Suppose visitors were randomly assigned and measurement is complete. The observed difference is .02. An illustrative large-sample standard error is sqrt(.1*.9/100 + .12*.88/100) ≈ .04423; a Normal approximation for the difference gives about [-.0667,.1067]. This approximation needs care with sparse counts and does not prove equality or a useful improvement.

Decision artifact: state the effect size that matters, planned stopping rule, collection checks, and uncertainty. Explain why the observed ordering alone cannot establish superiority. If the experiment was not randomized or visitors share sessions, state what changes in the analysis. The experiment-design lesson provides the review criteria.

2. Retrying during degradation​

Allow at most three total attempts, stopping on success. Under independent failures with constant probability p, expected attempts are 1+p+p². At p=.01,.1,.5 these are 1.0101,1.11,1.75. Eventual failure probabilities are .000001,.001,.125 respectively.

Now suppose one shared outage of probability .1 determines every attempt: all fail during the outage, all succeed otherwise. Expected attempts are .9*1+.1*3=1.2; eventual failure probability remains .1. Compare with 1.11 and .001 under independent per-attempt failures of .1. Retrying cannot create an independent opportunity when the failure condition persists.

Decision artifact: state the retry cap, backoff policy, load cost, deadline, and evidence needed for the independence assumption. A mathematically correct independent model is still the wrong production model when its assumptions fail.

3. A mean that hides slow requests​

A synthetic population has 98% of requests at 100 ms and 2% at 1,100 ms. The mean is 120 ms, the lower median and p95 are 100 ms, and the lower p99 is 1,100 ms. Here quantile q means the smallest t with F(t)>=q. State the convention because finite-sample percentile implementations can differ.

Decision artifact: show the two-point CDF, calculate the probability of exceeding 200 ms, and define an alert tied to the intended request population. Do not claim that a lower mean necessarily fixes a tail SLO. Sampling bias or missing timeout records can hide the slow requests entirely.

4. Rare fraud and finite reviewer capacity​

For 100,000 transactions, assume prevalence .002, sensitivity .95, and false-positive rate .01. The expected table has 190 true flags, 10 misses, 998 false flags, and 98,802 true negatives. Precision is 190/1188 ≈ .159933; expected review volume is 1,188. Sensitivity alone cannot determine either number.

Decision artifact: compare this operating point with sensitivity .90 and false-positive rate .002. The second point gives 180 true flags, 20 misses, 199.6 expected false flags, precision about .474183, and review volume 379.6. Fractional counts are expectations over repeated populations, not literal fractional transactions. Explain which costs and capacity constraints would determine the preferable choice; higher precision alone does not decide it.

5. Repeated observations and narrow error bars​

Collect one latency measurement from each of 20 independent sessions. A script duplicates each row 100 times and then reports 2,000 observations. The numerical average stays unchanged; treating duplicates as independent would divide the original estimated standard error by 10 without collecting new information.

Decision artifact: identify the true independent unit, explain the covariance of copies, and propose additional data collection. Use the original session observations or a justified model of within-session dependence. A larger file is not necessarily a larger experiment.

Review​

Submit the model, calculation, decision, and missing evidence for one case. Use practice solutions and the assessment map to identify the competency involved. A reviewer should alter one assumption and ask which conclusion survives.