Skip to main content

Averages, Simulation, and Confidence Language

What This Concept Is​

A sample mean is a random variable. Its behavior depends on how observations were collected, not just how many rows a file contains. Keep three statements separate.

Law of large numbers (LLN). For independent identically distributed (i.i.d.) observations with finite absolute expectation, the sample mean converges to the population mean. The weak law states convergence in probability. It does not promise that every additional observation moves the average closer, or give a universal adequate sample size.

Variance of an average. For independent observations with common finite variance sigma squared,

Var(mean) = Var((X1 + ... + Xn)/n)
          = (Var(X1) + ... + Var(Xn))/n²
          = sigma²/n.

This identity holds at every positive n. It does not require a Normal distribution or the CLT. The standard error is sigma/sqrt(n), so four times as many independent observations halve that standard error. With dependence, retain the covariance terms:

Var(mean) = [sum Var(Xi) + 2 sum(i<j) Cov(Xi,Xj)] / n².

Central limit theorem (CLT). For i.i.d. observations with finite, strictly positive variance, the distribution of sqrt(n)*(mean-mu)/sigma converges to the standard Normal distribution. It describes the limiting shape of a standardized average. It does not make the original observations Normal, supply a universal finite-n approximation error, or establish that 30 observations suffice. Quantitative error bounds require additional assumptions. See the textbook statement and derivation.

Why It Matters Here​

Engineering measurements contain several different uncertainties. A simulation has random numerical error conditional on its model. An experiment has sampling uncertainty and possible confounding. A model may be wrong about production behavior even when its simulation is extremely precise. Adding simulated trials cannot repair a missing shared failure mode.

Suppose you sample requests from one user session. Their latencies may share a network connection, cache state, or server. Treating those rows as independent can understate uncertainty. Define the sampling or randomization unit before choosing a standard error.

Concrete Examples​

Independent batches versus copied observations​

Let X be Bernoulli(.5), so Var(X)=.25. The average of 100 independent copies has variance .0025. A file containing the same realized X repeated 100 times has average X and variance .25 across repeated experiments. Both files have 100 rows. Their information content is different.

In Lab C, independent exponential observations have mean 1 and variance 1. Batch means have variance 1/n. The report compares this exact result with simulated values at n=1,5,30,100. The fraction of standardized batch means inside [-1.96,1.96] explores the Normal approximation; it is not a correctness test that must equal .95.

Zero observed failures does not establish zero risk​

With k successes in n independent Bernoulli trials, the familiar Wald interval is k/n ± 1.96*sqrt((k/n)*(1-k/n)/n). At k=0 it collapses to [0,0]. This unreasonable certainty is one reason to study a different interval procedure.

The Wilson score interval in the lab uses the formula supplied in its README. At k=0,n=20, it is approximately [0,.161125]. Its default is a nominal two-sided 95% procedure. Actual finite-sample coverage varies with n and the true probability; the code enumerates that variation. Wilson is not the exact binomial interval. NIST gives the Wilson formula and distinguishes it from exact binomial intervals.

For a fixed true probability p, coverage can be calculated without simulation:

coverage(n,p) = sum over k=0..n:
    indicator(interval(k,n) contains p) * C(n,k)*p^k*(1-p)^(n-k)

This is the probability that a fresh sample's interval covers the fixed parameter, under the stated model. Once observed, a particular frequentist interval is fixed. A Bayesian probability statement about a parameter requires an explicit prior and posterior model.

Planning Monte Carlo precision​

If a Monte Carlo estimator averages i.i.d. values Y with finite variance v, its standard error is sqrt(v/n). An approximate Normal 95% half-width w suggests n ≈ 1.96²*v/w²; this is a planning approximation, not a guaranteed error bound.

For a Bernoulli proportion, v=p(1-p) is at most .25. An absolute half-width .01 gives a conservative planning value near 9,604 trials using z=1.96. A 1% relative error at p=.01 means an absolute error .0001 and needs much more data. State which precision you mean. A rare event might never occur in a small simulation, making an estimated standard error misleadingly zero.

Common Confusion / Misconceptions​

  • "More rows remove bias." Biased sampling can converge very precisely to the wrong target. Increasing n addresses some random errors, not selection bias.
  • "CLT gives a finite-n guarantee." The basic theorem is asymptotic. Strong skewness, rare events, or heavy tails can make the approximation poor. Infinite variance violates this version's assumptions.
  • "A confidence interval must cover in exactly 95% of reruns." Even actual coverage depends on the procedure and model; a finite Monte Carlo experiment also fluctuates around that coverage.
  • "Overlapping intervals prove no difference." Analyze the difference directly. Its variance includes both marginal variances and any covariance; an equal-variance pooled formula is not universally appropriate.
  • "Bootstrap fixes any data." Resampling must respect the sampling unit and dependence. Sparse extreme tails and an unrepresentative sample remain problems.
  • "Check every minute and stop when the interval excludes zero." A fixed-sample procedure does not automatically retain its nominal properties after data-dependent stopping. Plan a fixed stopping rule or learn an appropriate sequential method.

How To Use It​

  1. Name the target population, quantity, sampling unit, and collection process.
  2. Derive the estimator's expectation and variance under explicit assumptions.
  3. Distinguish exact algebra, an approximation, and empirical evidence.
  4. Fix sample size and analysis before observing the result for this module's experiments.
  5. Check coverage on a model where the truth is known; compare enumeration with simulation.
  6. State what the data cannot establish, including causal effects or a population quantile from aggregate means.

For a simulated coverage estimate based on R independent reruns and true coverage c, the simulation standard error is sqrt(c*(1-c)/R). This quantifies noise in the coverage experiment, not uncertainty about a production failure rate.

Transfer / Where This Shows Up Later​

Benchmarking needs comparable workloads and independent experimental units. SLO estimation needs attention to rare events and sampling. Randomized algorithms need their own expectation or concentration proofs; the LLN alone does not prove quicksort's complexity. Later experiment design builds on this module's release-decision exercise.

Check Yourself​

  1. Which result gives Var(mean)=sigma²/n? Variance algebra plus zero covariances, not the CLT.
  2. Does the LLN make the next observation compensate for earlier bad luck? No.
  3. Can copying 20 measurements produce 2,000 independent trials? No.
  4. If a 95% procedure has actual coverage .92 at a particular n,p, what should repeated simulation approach? .92, conditional on that model.
  5. Does a narrow interval from a simulator validate the simulator's independence assumption? No.

Mini Drill or Application​

Use the supplied report to compare exact Wilson coverage and its Monte Carlo estimate at n=20,p=.01,.1,.5. Express each difference in simulation standard-error units. Run a second seed and report both outcomes, including surprises; do not keep only the closer result. Derive why repeated copies would invalidate that standard-error calculation. Submit your reasoning before consulting the practice solutions.

Read This Only If Stuck​

In Introduction to Probability, find the law of large numbers, central limit theorem, and inequalities. Use Mathematics for Computer Science for the probability-versus-confidence discussion. Follow the reading guide for a specific exercise rather than reading either book cover to cover.