Skip to main content

Worked Examples

Example 1: A Baseline That Changes the Decision​

A team proposes a transformer to prioritize support tickets. First define the decision: route each new ticket to one of six queues within two minutes. The current workflow resolves 71% without reassignment. A majority-class predictor reaches 38%; a reviewed keyword rule reaches 68%; logistic regression reaches 76%.

The transformer reaches 79%, but costs twelve times more per request and loses its advantage on the newest-month split. The correct conclusion is not “the transformer wins.” The linear model is the current candidate; the transformer needs temporal and cost evidence.

Rejected alternative​

Random splitting was rejected because repeated customers and near-duplicate replies crossed the boundary. The accepted split groups customers and holds out the latest month.

Example 2: Threshold Under Capacity​

A risk model emits calibrated probabilities. Reviewers can inspect 200 cases daily. Selecting 0.5 produces only 70 cases and misses costly events. Select the threshold from the precision-recall-capacity curve, then lock it before final evaluation.

Defense prompts​

  • Why is the manual workflow part of the baseline set?
  • Which evidence would overturn the model choice?
  • Where can preprocessing leak future information?
  • Why does a calibrated probability not settle the action threshold?

Source backbone​

Use the scikit-learn model evaluation guide and Designing Data-Intensive Applications selectively.