Measure an improvement without rewarding the wrong behavior
What This Concept Is
A decision experiment specifies a hypothesis, observation population, baseline, intervention, success signal, guardrail and reconsideration point before results arrive. Activity proves that work happened; an outcome describes what changed for the people or system the work was meant to help.
Why It Matters Here
Deployments, documents and platform users are observable, but increasing them can fail to improve reliability or delivery. The reliability module already showed how excluding rejected requests can make a service look healthy. Leadership can create the same distortion by choosing convenient team metrics.
Concrete Example
A self-service replay tool aims to reduce operator effort. Before the experiment, 40 eligible tasks took 120 total operator minutes. Afterwards 20 took 40 minutes. Mean recorded effort fell from three to two minutes per task, but total minutes alone exaggerates the change because volume halved. If the difficult tasks moved to a support queue excluded from measurement, even the per-task comparison is biased.
Record eligibility, task mix, collection completeness and end-to-end completion. Include failed attempts and escalations. A guardrail could stop the pilot after any duplicate business effect; another could bound unreviewed queue age. These are teaching choices to justify for the actual service, not universal thresholds. A tiny before/after sample suggests further investigation; it does not establish causation.
Common Confusion / Misconception
A service delivery metric is not an individual productivity score. Do not rank engineers by deployment count, compare unrelated services as if their constraints matched, or turn one number into a compulsory target. More activity can reflect rework. A pilot with no incidents provides limited evidence when exposure is small.
How To Use It
Choose one service boundary and preserve definitions across the comparison. Pair intended benefit with reliability, operator effort and affected-user safeguards. Record changes in traffic, staffing and task complexity. Establish an owner who can stop the experiment and a date to review it. If evidence is missing, mark the result unknown rather than filling in an optimistic value.
Check Yourself
Which tasks disappeared from the denominator? Can adoption be mandatory while usefulness remains low? What competing explanation fits the improvement? Would the team behave differently if this number determined rewards?
Mini Drill or Application
Design a measurement sheet with five raw example observations, including a failed task and a missing duration. Explain which summaries can and cannot be computed. Ask a reviewer to invent a way to improve your headline number while worsening user outcomes, then revise the guardrails.
Read This Only If Stuck
Fundamentals of Software Architecture — leadership, negotiation and effective teams. DORA's software delivery performance metrics explains service context and pitfalls in metric use. Read the current definitions when applying them; this exercise does not require building a complete DORA dashboard.