Evaluate evidence without manufacturing a pass
What This Concept Is
An evaluator transforms observations into a decision under a fixed contract. Measurement and evaluation are separate: a perfectly implemented evaluator can accept an incomplete dataset. Therefore its output is conditional on instrumentation coverage.
Why It Matters Here
Fitness checks need an explicit third state: insufficient evidence. Treating an empty stream as 100% success creates a green result when the collector is broken. Silently removing malformed observations creates the same problem at smaller scale.
Concrete Example
The downloadable evaluator selects attempts with start <= timestamp < end. A good attempt succeeds within the inclusive latency budget. A timeout remains in total even when completed latency is unknown. Compare integers: good × target denominator >= total × target numerator. Fractions avoid rounding a failing ratio into a passing percentage.
For a 3/4 target and four observations, three good passes, two good fails. Zero observations yields ratio None and status insufficient. One good observation with minimum=2 remains insufficient. Duplicated attempt IDs raise ValueError; they do not silently double a successful sample. Rows outside the window are still validated so corrupt records are not hidden by a chosen filter.
Common Confusion / Misconception
Unknown elapsed time for a timeout differs from an entirely missing attempt. The first is representable as a failed observation; the second cannot be detected without an independent expected-attempt ledger or collector coverage check. The evaluator does not invent missing rows.
How To Use It
Implement the README contract independently, predict edge cases, then run the harness. Keep input immutable. Before service integration, record intended attempts separately and reconcile terminal outcomes, including cancellations and timeouts, after the window closes. Define a bounded waiting policy for unfinished work. Do not treat a truncated export as a complete interval.
Check Yourself
Why is the right window boundary exclusive? Why reject booleans as integer timings in Python? Why validate observations that will not count? What extra evidence would reveal a collector dropping failures?
Mini Drill or Application
Supply three independent fixtures: boundary timestamps, low traffic, and a mixture of slow successes and fast failures. For each predict good, total, ratio and decision. Inject a duplicate ID and a successful row with no latency; require a visible error. Then append one failed attempt and prove the good count cannot increase.
Read This Only If Stuck
Fundamentals of Software Architecture — fitness functions and architecture measurements. Read the lab README before implementation; open reference.py only after retaining your independent attempt.