Skip to main content

Quality evidence assessment and calibration

Attempt before reading calibration. Score each gate 0 absent/incorrect, 1 correct with substantial prompting or incomplete evidence, 2 independent correct work with a counterexample and limitation, 3 transfers correctly to a fresh scenario. Every gate must reach 2; a strong average cannot compensate for an integrity or evidence gap.

Required gates​

  • QA1 — Scenario contract: operationalize one normal-load and one failure scenario, including eligible population, measurement boundary, threshold, owner and missing-data behavior.
  • QA2 — Denominator: hand-predict mixed successes, slow responses and timeouts; explain why success-only data changes the claim.
  • QA3 — Evaluator: implement the archive contract and pass checks; add three independent fixtures, including duplicate identity and an empty window.
  • QA4 — Measurement coverage: reconcile a planned-attempt ledger with observations. Remove one failed row deliberately and show that a separate coverage check detects it; explain why the evaluator alone cannot.
  • QA5 — Tactic comparison: run two implementations under the same scenarios with repeated measurements, cohort results and one held-out failure trace. Preserve an independent correctness invariant and report costs.
  • QA6 — Architecture decision: align the measurement boundary with runtime/deployment views; explain a rejected option, assumption, evidence limit and reversal trigger. Demonstrate an actual change to test one modifiability proxy.

Fresh defense and repair​

Without reference code, analyze a changed target, a request at the window boundary, a missing cohort and a failed collector. Modify the evaluator or contract only when justified. After at least 48 hours, use an unseen dataset and a different failure scenario. Record mistaken predictions and repair the weakest gate before advancing. Passing reference tests does not demonstrate independent mastery. Disclose tools, reference use and other assistance.

Calibration — open after attempting​

Three successful 100 ms requests and one timeout meet 3/4 with budget=100. At budget=99 none are good. With minimum=5 the four-observation result is insufficient even if all are successful. No observations yields ratio None, never perfect reliability. A timestamp equal to end is excluded; one equal to start is included. Duplicate attempt IDs are invalid input under this lab contract.

A collector dropping the timeout leaves an apparently perfect dataset; the evaluator cannot know it is incomplete. Reconciliation against four scheduled attempt IDs exposes the missing outcome. Real clients need their own coverage and terminal-outcome policy.

A pooled 100/104 result conceals a zero-of-four cohort. Whether this violates the product requirement depends on the predeclared cohort policy. Do not invent a new policy after choosing a winner. A cache that returns a deleted or unauthorized document fails its correctness gate even if every request is fast.

A dependency rule can protect a boundary while a change still requires many teams or deployments. Keep the runnable boundary check and a concrete change exercise; neither alone proves broad modifiability.