Observe the observer during an incident
What This Concept Is
Service health and telemetry delivery health are separate signals. Missing observations can result from dropped buffers, collector failure, sampling or a broken pipeline.
Why It Matters Here
The original alert/runbook drill must distinguish quiet service from silent instrumentation.
Concrete Example
Stop a collector while sending synthetic requests. The service remains responsive but the dashboard freezes. A heartbeat or ingestion-freshness check should identify this gap. Sampled traces may help diagnose a request but cannot automatically serve as an unbiased denominator for request-success SLOs.
Common Confusion / Misconception
No alerts is not proof of no failures. A trace ID in two records does not prove the intermediate steps are complete or causally correct.
How To Use It
Record emitted, accepted and observed counts where feasible, sampling policy, queue limits and exporter failures. Exercise collector outage and recovery, checking whether lost data is explicitly reported rather than retroactively invented.
Check Yourself
Which signal reports collection failure? Does sampling preserve your SLI population? Can timestamps across hosts establish exact order?
Mini Drill or Application
Repeat the original latency incident with the collector unavailable. Write what can still be concluded from client evidence and what remains unknown.
Read This Only If Stuck
Building Secure and Reliable Systems provides selective operational reasoning. Use the module sources below for current tool behavior; do not read a whole book before attempting the exercise.
OpenTelemetry sensitive data; Collector security; Google SRE monitoring.