Skip to main content

Variance, Joint Structure, and Covariance

What This Concept Is​

Expectation tells you where a distribution is centered. Variance tells you how spread out it is around that center. [ \mathrm{Var}(X) = E[(X - E[X])^2] = E[X^2] - (E[X])^2. ] The second form ("mean of the square minus square of the mean") is the computational workhorse. The standard deviation ( \sigma_X = \sqrt{\mathrm{Var}(X)} ) has the same units as ( X ) and is usually the quantity reported -- variance is mathematically cleaner but in units of "( X ) squared."

When multiple random variables interact, covariance measures whether they tend to move together: [ \mathrm{Cov}(X, Y) = E[(X - E[X])(Y - E[Y])] = E[XY] - E[X] E[Y]. ] Positive covariance means that when ( X ) is above its mean, ( Y ) tends to be above its mean as well. Negative covariance means the opposite. The correlation ( \rho_{XY} = \mathrm{Cov}(X, Y) / (\sigma_X \sigma_Y) ) is the dimensionless, unit-free version, always between ( -1 ) and ( 1 ).

A few identities that drive almost all variance computations:

  • ( \mathrm{Var}(aX + b) = a^2 \mathrm{Var}(X) ). Shifts do not change variance; scaling squares.
  • ( \mathrm{Var}(X + Y) = \mathrm{Var}(X) + \mathrm{Var}(Y) + 2, \mathrm{Cov}(X, Y) ).
  • If ( X, Y ) are uncorrelated, ( \mathrm{Var}(X + Y) = \mathrm{Var}(X) + \mathrm{Var}(Y) ).
  • For independent ( X_1, \dots, X_n ), ( \mathrm{Var}(\sum X_i) = \sum \mathrm{Var}(X_i) ).

The key concentration inequalities use variance directly:

  • Markov: ( P(X \ge a) \le E[X]/a ) for non-negative ( X ).
  • Chebyshev: ( P(|X - E[X]| \ge k \sigma_X) \le 1/k^2 ). Variance -> tail bound.

Why It Matters Here​

In systems and data work, the mean alone is almost never a complete description. Two algorithms can have the same expected latency but wildly different reliability. A service with mean latency 100 ms and std dev 10 ms is fundamentally different from one with mean 100 ms and std dev 200 ms -- the second routinely goes to 500 ms and is incompatible with a p99 SLO.

Covariance matters when:

  • variables are not independent, so variances of sums do not just add;
  • totals are formed from correlated pieces (a service's total error count is sum over components, and components share failure domains);
  • you need to know whether noise cancels or compounds.

This is also the conceptual door to portfolio-style thinking: if you replicate a noisy computation 10 times with independent noise, the variance of the average is ( \sigma^2 / 10 ). If the noise is fully correlated (e.g., shared clock drift), the variance of the average is ( \sigma^2 ) -- no benefit at all. This distinction is why i.i.d. simulation works and why over-simplified redundancy analysis fails.

Concrete Examples​

Example 1 -- Same mean, different variance. Two games:

  • Game A: wins $2 with probability 1/2, loses $0 with probability 1/2.
  • Game B: wins $100 with probability 1/2, loses $98 with probability 1/2.

Both have ( E[\text{payoff}] = 1 ). But [ \mathrm{Var}(A) = (2-1)^2 \cdot \tfrac{1}{2} + (0-1)^2 \cdot \tfrac{1}{2} = 1, ] [ \mathrm{Var}(B) = (100-1)^2 \cdot \tfrac{1}{2} + (-98-1)^2 \cdot \tfrac{1}{2} = 9801. ] Standard deviations: ( \sigma_A = 1 ), ( \sigma_B = 99 ). Same average reward, hundred-fold difference in volatility. If you played Game B 100 times, the variance of the sum is ( 100 \cdot 9801 ) (assuming independence), and the variance of the average is ( 9801/100 = 98.01 ); the average over 100 plays has standard deviation ( \approx 9.9 ), still much larger than the mean. Game A is stable; Game B is a lottery that happens to have the same expected value.

Example 2 -- Covariance in correlated server loads. Two services A and B both spike during the same traffic surge. Say ( X ) = A's load, ( Y ) = B's load, each with ( E = 100 ), ( \sigma = 40 ), and ( \mathrm{Cov}(X, Y) = 0.8 \cdot \sigma^2 = 0.8 \cdot 1600 = 1280 ) (correlation 0.8). Then the total load ( T = X + Y ) has [ \mathrm{Var}(T) = \mathrm{Var}(X) + \mathrm{Var}(Y) + 2,\mathrm{Cov}(X, Y) = 1600 + 1600 + 2560 = 5760, ] so ( \sigma_T = \sqrt{5760} \approx 75.9 ). If the services were independent, ( \mathrm{Var}(T) = 3200 ) and ( \sigma_T \approx 56.6 ). Correlation inflates the total variance by a factor of ( 1.8 ), corresponding to 34% larger ( \sigma ). In capacity planning, ignoring this correlation means under-provisioning headroom -- a very common operational mistake. Conversely, if the services were negatively correlated (e.g., one takes load when the other is down), ( \sigma_T ) would be smaller, not larger.

Common Confusion / Misconceptions​

"Variance is 'probability of being wrong.'" Variance is the expected squared deviation from the mean. It has nothing directly to do with the probability of any event; it is a spread summary.

"Variance and standard deviation carry the same information so they are the same thing." Mathematically related, but the units are different: if ( X ) is in seconds, ( \mathrm{Var}(X) ) is in seconds² and ( \sigma_X ) is in seconds. Always report ( \sigma ) in production alongside the mean; it is directly comparable to the mean.

"Zero covariance implies independence." False in general. Independence implies zero covariance, but two variables can be fully deterministic in each other (( Y = X^2 ) when ( X ) is symmetric about 0, for example) and still have zero covariance. Covariance captures only the linear part of the relationship.

"Correlation of 0.3 is 'low.'" In engineering, even moderate correlations (0.2 - 0.4) can meaningfully inflate variance of sums. "Low correlation" is a dangerous relative term.

"( \mathrm{Var}(X + Y) = \mathrm{Var}(X) + \mathrm{Var}(Y) ) always." Only when ( \mathrm{Cov}(X, Y) = 0 ). For independent variables, yes. For correlated variables, the covariance term can easily dominate.

"Variance of a sample always estimates population variance well." The sample variance ( s^2 = \frac{1}{n-1} \sum (X_i - \bar X)^2 ) is unbiased for the population variance, but it is a noisy estimator: its own variance is large for small ( n ). Reporting a variance from 10 samples is almost meaningless.

How To Use It​

Use variance when you need to talk about stability, spread, or tail risk. Use covariance when a sum or pair of variables may share structure.

Operational checklist:

  1. Compute the mean first. Without ( E[X] ), you cannot compute ( \mathrm{Var}(X) ).
  2. Compute ( E[X^2] ) (often via LOTUS) and subtract ( (E[X])^2 ).
  3. Report standard deviation, not variance, when communicating with humans.
  4. For a sum, explicitly check whether terms are correlated. If yes, include the ( 2 \sum_{i<j} \mathrm{Cov}(X_i, X_j) ) term.
  5. Use Chebyshev to turn a variance bound into a tail bound: ( P(|X - \mu| \ge k \sigma) \le 1/k^2 ). Chebyshev is loose but distribution-free.
  6. For sharper bounds on sums of independent variables, use Chernoff (Semester 2 material) -- but note that Chernoff requires independence, while Chebyshev needs only variance.
  7. Sanity-check with a known case: variance of Bernoulli(( p )) is ( p(1-p) ), max at ( p = 0.5 ); variance of ( \mathrm{Bin}(n, p) ) is ( np(1-p) ); variance of Poisson(( \lambda )) equals ( \lambda ).

Transfer / Where This Shows Up Later​

  • Semester 2 (Randomized algorithms). Chebyshev and Chernoff bounds are the main tools for "with high probability" results. Chebyshev uses variance; Chernoff uses the moment generating function (built from higher moments than variance).
  • Semester 2 (Hashing). The variance of the number of items in a single bucket (Bernoulli sum) is ( n \cdot (1/m)(1 - 1/m) ); this is used to bound max bucket load via Chebyshev when independence is limited.
  • Semester 5 (Queueing). Variance of waiting times is typically larger than the mean squared -- "latency has heavy variance." The variance-to-mean ratio is a diagnostic for the arrival-process regularity.
  • Semester 6 (Distributed systems). Replication lag variance across replicas is the quantity that drives tail consistency -- even if mean lag is small, high variance means long-tail reads will occasionally see stale data.
  • Semester 6 (Linear algebra connection). Covariance matrices generalize pairwise covariances into a positive semidefinite matrix ( \Sigma_{ij} = \mathrm{Cov}(X_i, X_j) ). In Semester 6 and beyond, PCA decomposes this matrix into principal components -- the directions of maximum variance -- and is used to compress high-dimensional random-vector data.
  • Semester 8 (SRE). The 99th-percentile latency depends on the tail of the distribution, which is bounded via variance through Chebyshev and more tightly via Chernoff or empirical percentiles. "Reducing tail latency" is operationally "reducing variance of the latency random variable."
  • Semester 9 (Experimentation). Statistical power of an A/B test is inversely related to ( \sqrt{\mathrm{Var}/n} ). Halving the variance of the outcome (via better instrumentation or stratification) has the same effect on power as quadrupling sample size.

Check Yourself​

  1. Why can two distributions with the same mean behave very differently operationally?
  2. What does positive covariance mean in plain language? What about negative?
  3. Why does zero covariance not automatically imply independence? Give an explicit counterexample.
  4. Prove ( \mathrm{Var}(aX + b) = a^2 \mathrm{Var}(X) ) from the definition.
  5. Two coins are flipped, correlated so that both heads has probability 0.4, both tails has probability 0.4, and HT or TH has probability 0.1 each. Compute ( \mathrm{Var}(X + Y) ) where ( X, Y \in {0, 1} ) are indicators of heads. Compare to the independent case.
  6. State Chebyshev's inequality and use it to bound the probability that a ( \mathrm{Bin}(100, 0.5) ) deviates by more than 20 from its mean. (Answer: ( \sigma^2 = 25 ), so ( P \le 25/400 = 0.0625 ). This is loose; the true probability is ( \sim 10^{-5} ) by CLT. Chebyshev is distribution-free but loose.)

Mini Drill or Application​

For each situation, say whether variance or covariance is the main quantity to inspect, and why:

  1. Comparing two retry strategies with the same average latency but different shapes.
  2. Total load from two services affected by the same daily traffic pattern.
  3. Stability of a random hash-bucket occupancy count across independent trials.
  4. Estimating user engagement across a cohort of users whose behaviors are independent.
  5. A portfolio of three redundant replicas that share a power domain -- how much does the shared failure domain matter for the variance of "number of replicas up"?

Simulation drill -- variance reduction by averaging. In Python, simulate ( n = 10{,}000 ) draws from ( \mathrm{Bernoulli}(0.5) ) and compute the sample variance of single draws (should be near 0.25) and of the sample mean of 100 independent draws (should be near 0.25/100 = 0.0025). Then simulate a correlated version: use the same Bernoulli outcome for each of the 100 draws (complete correlation). The variance of the mean should now be 0.25, not 0.0025 -- averaging correlated noise gives no variance reduction. This is the core of the i.i.d.-vs-correlated distinction that drives the Central Limit Theorem in Cluster 5.

Read This Only If Stuck​