Skip to main content

Splits, Leakage, and Evaluation Contracts

What This Concept Is​

The evaluation contract freezes population, unit of independence, time boundary, labels, exclusions, metrics, and split method before model selection. Treat every claim as an engineering claim: name the population, assumptions, versioned inputs, and observable result. The objective is not merely to obtain a number but to establish when that number is trustworthy and when it must not drive action.

Why It Matters Here​

This concept is part of the evidence chain from problem framing to a defensible baseline. Weakness here contaminates later model comparisons, safety review, and production monitoring. A learner must be able to explain the mechanism, implement it, and show evidence that an independent reviewer can reproduce.

Concrete Example​

A random ticket split leaks repeated customers and future vocabulary. A time-based split with customer grouping better simulates deployment. The team records the decision before inspecting final-test performance, runs the comparison from a clean environment, and stores both the result and the rejected alternative. This makes the conclusion falsifiable rather than retrospective.

Common Confusion / Misconception​

Leakage is not limited to putting the label in a feature; preprocessing, duplicates, post-outcome fields, and tuning on the test set also leak. Another common error is to treat a library default as a methodological decision. Defaults may be reasonable starting points, but the report must connect each important choice to the deployment context and expected failure cost.

How To Use It​

Draw the data timeline, identify when every feature becomes available, hash near-duplicates, and quarantine the final test set.

Use this operating sequence:

  1. State the decision and assumptions before implementation.
  2. Build the smallest reproducible experiment that could disprove the claim.
  3. Compare against a credible alternative under identical conditions.
  4. Inspect failures and important slices, not only the aggregate score.
  5. Save code, configuration, data fingerprint, and result as one evidence bundle.
  6. Record a stop condition and the next action if evidence fails.

Check Yourself​

  1. Which assumption in this concept is easiest to violate silently?
  2. What artifact would let a reviewer detect that violation?
  3. Which alternative must be compared before accepting the result?
  4. What deployment change could invalidate today’s conclusion?

Mini Drill or Application​

Apply the operating sequence to a small tabular or text dataset. Produce a one-page evidence note containing the claim, method, result, one failure example, one rejected alternative, and a go/limit/stop conclusion. Pair-review the note without showing the implementation first; if the reviewer cannot identify what was tested, rewrite the contract.

Read This Only If Stuck​