Skip to main content

Capstone test oracles and defect evidence — assessment

Score 0 absent/incorrect, 1 prompted/incomplete, 2 independently supported with explicit limits, 3 justified fresh transfer. Every category requires at least 2; a high average cannot hide a missing category.

Required evidence​

  • CT1 — Vertical slice: Retain actual client/API/datastore path, deployed or local runtime identity and observable success/failure.
  • CT2 — Oracle: Independently derive expected results and detect a relevant faulty variant.
  • CT3 — Reproduction: Provide a reproducible defect packet and explain the discriminating experiment.
  • CT4 — Regression: Show failure before, fix, success after and a justified remaining limitation.
  • CT5 — Boundaries: Retain real integration and contract evidence for the highest-risk mocked assumption.
  • CT6 — Maintainability: Retain original static/dependency/coupling review and a repaired issue tied to a risk, not cosmetic metrics.

Calibration​

A test that expects the output of the same function under another name is not independent. A retry-until-green run is not a regression fix. A real datastore plus a controlled concurrent schedule gives evidence about that setup, not every possible deployment. A seeded fault must be labeled.

Delayed transfer and integrity​

After at least 48 hours, have a reviewer change one assumption or select a previously unseen failure. Predict the outcome, collect fresh evidence and explain discrepancies without the calibration text. Preserve mistakes, repairs, collaboration and generated-assistance disclosures. Record unavailable tools or reviewers as pending evidence, never fabricated completion. Author reference checks are not learner runtime experiments.