Skip to main content

Regression evidence and test sensitivity

What this concept is​

A regression test isolates an observable failure and protects the intended repair. Test sensitivity means the test actually detects a relevant incorrect implementation. A passing suite without a failing demonstration may be exercising the wrong path.

Why it matters here​

Caches and mutable aliases create failures that isolated happy-path calls miss. Review must follow sequences and ownership, connecting the runtime and composition contracts from Modules 3–4 to a complete feature.

Concrete example​

Fix the directory filter but keep a cache keyed only by id. Request (a,7) then (b,7): the second call can still return Ada. A miss can also poison the cache: (a,8) returns None, but (b,8) must return Bea. Next mutate the returned name and request the record again. Under the fresh-result contract, that mutation must not affect the directory. These are three different claims and need distinguishable evidence.

Common confusion / misconception​

Do not calculate expected outputs by calling the implementation under test. That repeats its assumptions. A failure caused by an import error does not prove a regression was reproduced. Exhaustive enumeration over a small fixture is useful but does not prove safety for arbitrary databases, threads or malformed input.

How to use it​

Write the expected result directly from the contract. Retain the failing assertion and input, apply one repair, then rerun the regression and adjacent behavior checks. Try temporarily restoring the defective filter or cache key: the relevant test should fail. Keep this fault injection in a disposable copy. Prefer public behavior assertions to checks that demand a particular internal dictionary.

Check yourself​

Would the cache test fail with caching removed entirely? It should not: no cache is a valid implementation. Why test a miss followed by a hit? Which test distinguishes constructor snapshot ownership from returned-result ownership?

Mini drill or application​

Add a fresh three-call sequence with overlapping ids. Explain expected results without running code. Add a caller-mutation test and show that it fails for a returned internal dictionary. Then restore the repair and keep the actual output.

Read this only if stuck​

Read the tests section of Google's review guide and selective test-design discussion in Good Code, Bad Code. The exercise checks contract sensitivity rather than coverage percentages.