Skip to main content

From quality scenarios to evidence contracts

What This Concept Is​

An evidence contract states which observation could refute a quality claim. A scenario supplies source, stimulus, environment, artifact, response and measure. The contract adds population, measurement boundary, decision rule, owner and missing-evidence behavior. These additions are this curriculum's operational checklist, not a claim that every architecture method mandates this exact template.

Why It Matters Here​

An architecture experiment can pass while testing the wrong promise. A fast successful response says nothing about failed requests discarded before computing latency. The work connects Semester 06's invariants to Semester 07's quality scenarios: correctness conditions remain mandatory even when performance improves.

Concrete Example​

A synthetic document service handles an agreed mix of reads during a fixed test interval. Measure every eligible attempt at the client boundary, from dispatch to terminal response. The teaching contract says at least 3/4 of attempts must succeed within 100 ms. This deliberately small target makes fixtures readable; it is not a recommended production SLO.

Three successful observations of 100 ms and one timeout give good=3, total=4. The result meets 3/4 exactly. Filtering the timeout gives 3/3 and hides the harm. A successful 101 ms response is also bad under this contract. A request dispatched exactly at the end belongs to the next window. Freeze the window before examining results; do not slide it to remove a failure.

Common Confusion / Misconception​

A minimum sample count does not establish statistical confidence. A synthetic test does not establish a month's reliability. Attempt-based success is not user-journey success: retries alter the denominator. Name which one the product needs before choosing instrumentation.

How To Use It​

For each top-three scenario, write the eligible population, start/end boundary, terminal outcomes, threshold and handling of no data. Keep an explicit integrity invariant such as no cross-tenant reads. A good latency score cannot compensate for violating that invariant. Document which assumptions your experiment cannot test.

Check Yourself​

Why does a successful slow attempt count as bad? What is lost when only server-side completed requests are observed? Can a target of zero pass with no observations?

Mini Drill or Application​

Rewrite “search should be fast” for normal load and a cache outage. Specify the request mix, dataset, measurement point and missing-data action. Identify one alternative denominator that changes the decision. Submit the contract before collecting measurements.

Read This Only If Stuck​

Fundamentals of Software Architecture — revisit measurable characteristics. SEI Quality Attribute Workshops provides the stakeholder scenario method. Implementing SLOs supplies the good-events/total-events perspective; this exercise's numeric policy is original.