Benchmark correctness and measurement boundaries
What this concept is
A benchmark is an experiment whose output must remain correct and whose timed region must match the claim. A faster result obtained by computing less is not evidence of a better implementation.
Why it matters here
Initialization, process startup and output can dominate a small loop. Compiler transformations can also remove or reshape work. The lab makes checksums observable, keeps initialization outside timing and inspects compiled kernels.
Concrete example
The harness initializes each cell with index modulo 97 and independently sums those values. Every measured traversal must match that checksum for every repetition. It uses separate translation units without LTO and consumes all results. It warms both paths and alternates which runs first over eight samples. The timer measures process CPU time, not wall-clock latency.
Common confusion / misconception
A volatile array would change optimization and memory-access semantics, so it is not a neutral fix for benchmark elimination. Sanitizer binaries are for correctness evidence, not performance comparison. Zero-duration measurements indicate inadequate resolution, not infinitely fast code.
How to use it
Run correctness checks first. Record compiler/version/flags and target. Inspect assembly for loops, vector instructions and calls. Choose a repeated workload with measurable duration; preserve every sample. Record background load and power conditions without inventing missing details. Keep the same arithmetic and data across comparisons.
Check yourself
Why is the independent checksum stronger than comparing the two kernels only to each other? Which costs are excluded by placing initialization before the clock? Why must CPU seconds not be labeled elapsed latency?
Mini drill or application
Change one traversal to skip a row and show that checksum validation rejects it. Then restore correctness and compare O0/O2 assembly without treating a larger speed difference as a better learning outcome.
Read this only if stuck
Computer Organization and Design for the hardware/software boundary. For x86 targets, use Intel’s manuals selectively to identify instructions; choose target-appropriate documentation on other hosts.