Skip to main content

Common statistical-learning bridge

Required before this semester's P4 AI-risk decision, including for learners who do not select AI Engineering. Begin after Semester 01 probability/inference and linear algebra and independent Python programming; the earlier retrieval benchmark is useful preparation. This is statistical evaluation literacy, not a deep-learning course.

20–30 additional focused hours: selected concepts and prerequisite repair 5–7, split/baseline experiment 7–11, error analysis and regularization 4–6, independent assessment/delayed repair 4–6. Semester 08 becomes 510–725 hours. Existing P4 engineering-assistance and policy-boundary work remains separately budgeted. These estimates require pilots.

Reuse the existing specialization lessons​

Read only these selected lessons, then complete the concrete synthesis below. Full specialization modules, neural networks, model-serving stacks and paid tools are not prerequisites.

These existing pages remain the concept source; this bridge supplies a bounded common task, original worked reasoning and SL1–SL4. Later AI-specialization students may reuse the matching artifacts, but must complete its remaining lessons, practice and transfer. This bridge is not credit for the entire specialization.

Lesson 1: the deployment question determines the split​

Suppose we predict an outcome for a previously unseen customer using information available at the first visit. Each customer has two recorded visits and a stable outcome label. A row split can place one visit in training and the other in test. An ID lookup can then appear excellent without learning a rule for new customers. The split is not intrinsically meaningless: it may estimate another question about known customers. It is wrong evidence for our stated new-customer question.

Define the prediction time, target, population, unit of independence and fields available then. A refund-issued field recorded after the outcome is not an admissible predictor for the earlier decision, however predictive it looks. A pipeline cannot repair a feature whose meaning leaks the answer. If deployment concerns future periods, a time-respecting split may be needed; grouping alone does not address temporal drift.

Download the evaluation counterexample. Python 3.9+ and the standard library suffice. Eight synthetic entities have two visits each and labels equal to entity parity. The reference compares overlapping entities with disjoint training entities 0–3, validation 4–5 and test 6–7. The lookup gets 100% on overlapping entities. On validation, lookup and a training-majority baseline tie at 50%; a predeclared simplicity rule selects majority, then final test accuracy is 50%.

This result is deliberately constructed, not an estimate of real model quality. Four test rows represent only two entity groups. The public reference exposes all labels; it demonstrates a workflow rather than protecting a secret benchmark. Do not claim independent model evaluation from reproducing it. Before reading results, write your prediction and identify the leakage mechanism.

Lesson 2: a baseline and a metric define what improvement means​

A baseline must be fit using training information under the same task and split. For this fixture, choose the training-majority label with a predeclared zero tie rule. A lookup uses memorized training labels and falls back to that same majority for unseen entities. Test labels must not set the fallback or break ties. The reference intentionally retains the training-only model after validation selection; alternative refitting policies must be declared before test evaluation.

Keep true positives, false positives, false negatives and true negatives alongside summary metrics. For 100 cases with only five positives, predicting zero everywhere yields 95% accuracy and zero recall. Precision is undefined when there are no predicted positives, not proof that every emitted positive was correct. The reference represents undefined metrics as None. If there are no actual positives, recall is undefined too; report population counts and do not silently score missing slices as perfect.

Decision cost is separate from fit quality. With a fictional false-negative cost of 10 and false-positive cost of 1, compare total cost 10*FN + FP on the same population. A threshold chosen to minimize validation cost is not justified for every deployment prevalence or cost structure. Freeze that choice before final test. Inspect group-specific failures and sample counts; do not infer fairness from one aggregate number or invent confidence from two groups.

Exercise: labels [1,0,1,0], predictions [1,1,0,0] give one of each confusion outcome, accuracy/precision/recall 1/2 and cost 11. Derive these by hand before using the metric helper. Then evaluate an all-negative predictor and explain why a division-by-zero result must be explicit.

Lesson 3: control capacity without consulting the final test​

To see regularization directly, fit a one-feature model prediction = w*x, with no intercept, to x=[1,2], y=[1,2]. Define the objective as the SUM of squared residuals plus lambda*w*w, for lambda at least zero. Differentiating gives 2*w*sum(x*x) - 2*sum(x*y) + 2*lambda*w = 0. Therefore w = sum(x*y)/(sum(x*x)+lambda); here that is 5/(5+lambda).

At lambda=0, w=1 and training error is zero. At lambda=5, w=1/2 and the squared residual sum is 1.25. The penalty changes which fits are preferred; it does not guarantee better predictions on new data. On a validation point x=3,y=3, unregularized error is zero and regularized error is 2.25. Change validation y to 1.5 and that preference reverses. Choose using the declared validation criterion, not an assumption that regularization must always win. These are exact worked cases, not a realistic training study.

A mean-squared-error objective instead of a summed one changes the effective penalty scaling. Feature units also affect the meaning of a penalty. Fit scaling/imputation/feature-selection transformations only from training data, reuse them for validation/test, and refit them inside each cross-validation fold. For this bridge the scalar derivation has no preprocessing; do not add an unnecessary library installation merely to demonstrate the arithmetic.

Repeatedly choosing features, thresholds or penalties after viewing final-test performance turns that set into development data. More decimal places do not restore independence. Freeze the decision record, report the contamination honestly and reserve fresh evidence for the next claim. Generalization is an empirical question under assumptions; prediction alone does not establish causal benefit from an intervention.

Required synthesis and learner evidence​

  1. Write a deployment contract and feature-availability ledger. Explain the entity/time split you need and at least one excluded post-outcome field.
  2. Independently implement the majority baseline, ID lookup, split validator and confusion counts. Use the downloadable experiment only after an attempt. Preserve the failing row-split claim, corrected group split and limitations.
  3. Create a changed fixture with at least 12 entities, both classes and unequal visits per entity. Freeze training/validation/test group IDs and tie/selection rules before inspecting final labels. Report row and entity counts, per-slice errors and how unequal visits affect row-weighted versus entity-weighted metrics. Do not tune until a preferred model wins.
  4. Derive and check the scalar regularization example, then change one training pair and one validation target. Preserve rejected penalty choices. A hand calculation plus a short independent script is sufficient.
  5. Submit code, data/split fingerprint, predictions, full metric output, decision record and assistance declaration. State what additional evidence would be required for a deployment or causal claim. A correct decision not to use a learned model can pass.

If a reviewer supplies genuinely unseen cases, record their scope; otherwise label the packet self-assessed. Do not pretend a public calibration dataset is held out from the author. Sampling uncertainty, correlated observations and distribution shift remain limitations rather than inconveniences to hide.

SL1–SL4 assessment and calibration​

Use 0 absent/invalid, 1 assisted/incomplete, 2 independently correct with evidence and limits, 3 justified changed-case transfer. All four require at least 2 in addition to existing module and strand gates.

  • SL1: task/split integrity. Prediction time, admissible fields, group/time rationale and disjointness evidence. Calibration: a row split sharing entities does not establish unseen-entity performance; an outcome-only field is invalid regardless of score.
  • SL2: baselines and metrics. Training-only fits, fixed tie rule, validation selection, confusion counts and undefined cases. Calibration: the reference chooses majority at a validation tie; four test rows are not four independent customers. Accept other defensible predeclared tie rules with correctly scoped claims.
  • SL3: capacity and decision discipline. Correct scalar derivative/penalty calculation, changed-case selection and frozen final evaluation. Calibration: w=5/(5+lambda), training squared error 1.25 at lambda=5; regularization does not guarantee improvement. No test-driven retuning disguised as validation.
  • SL4: error analysis and transfer. Changed unequal-visit fixture, per-slice counts, weighting explanation and an honest use/no-use recommendation. Calibration: row averages give frequent visitors more weight; a rare-slice percentage without its denominator is insufficient. Prediction accuracy does not prove intervention benefit.

After 7–14 days redo a split critique and a metric/regularization calculation on changed inputs without notes. Active review time is counted above; waiting is not. Repair failed dimensions independently, preserving original attempts. Learner timing and external technical review remain pending.

Selective resources​

An Introduction to Statistical Learning provides author-hosted Python/R editions and course links. Select statistical learning, classification, resampling and regularization sections to repair a named gap; full-book reading and purchase are unnecessary. Use the scikit-learn common pitfalls guide for training-only preprocessing and leakage when moving to a library pipeline. Neither source is copied into this curriculum. Sources checked 2026-10-23; record installed package versions if you choose optional library work.