Exercises

Core

1. Rewrite a vague “predict churn” request as a decision contract.
2. Implement majority, rule, and linear baselines behind one interface.
3. Construct a synthetic leakage feature and demonstrate its misleading gain.
4. Compare random, grouped, and temporal splits on repeated entities.
5. Calculate accuracy, balanced accuracy, precision, recall, F1, and expected cost by hand for one confusion matrix.
6. Plot and interpret a reliability diagram.
7. Select thresholds for two different capacity budgets.
8. Create a dataset card and identify one use the data cannot support.
9. Prove offline/online feature parity on replayed records.
10. Reproduce a peer’s run from a clean environment.
11. Label 30 errors into a mechanism-based taxonomy.
12. Write a model-selection memo that rejects the highest-scoring candidate.

Transfer

Given a new domain and incomplete requirements, identify the five questions that must be answered before touching the data. Defend which work should remain non-AI.