Skip to main content

Evaluate Suggestions with Independent Tests

What this concept is​

An evaluation compares candidate behavior with explicit criteria on chosen cases. A schema check asks whether a suggestion has the expected form. A behavioral check asks whether it satisfies the task. A held-out case is reserved from the examples used to choose or revise the candidate.

Why it matters here​

AI-assisted work becomes inspectable when “this looks good” becomes “these cases passed, these failed, and these limitations remain.” You need not build a machine-learning benchmark to start.

Concrete example​

The lab represents a proposal as three fields: threshold, inclusive and unique. All three candidates are valid schema instances. The contract counts every session at least 25 minutes. On four independently specified cases, the strict-boundary candidate passes 2, the deduplicating candidate passes 3 and the contract-matching candidate passes 4.

Those are scores on authored fixtures, not measured accuracy for a model. A 4/4 score does not establish correctness on all inputs. Add a new case such as [25, 1, 25, 240] with expected 3 before deciding whether the candidate meets your purpose.

Common confusion / misconception​

Do not change an expected result to agree with the candidate. Do not call a case held-out after using its answer to tune the candidate. Do not execute text to make an evaluator “flexible”: this lab interprets a tiny validated data schema instead.

How to use it​

Write criteria and expected outputs first. Include ordinary, boundary, empty and repeated-value cases. Store failures with case index, expected and actual result. Keep schema errors distinct from wrong predictions. Add one unseen case and state what your suite does not cover. Record model/version and settings only when real model output was collected.

Check yourself​

Does valid JSON establish a correct suggestion? No. Syntax, schema and semantics are different checks. Does rejecting an extra instruction field prove a language model is injection-resistant? No; it only tests this parser.

Mini drill or application​

Implement the evaluator, then deliberately make it always report success and confirm a negative control catches the defect. Restore it and explain why evaluating the evaluator matters.

Read this only if stuck​

Google LLM introduction supplies limitations context; Anthropic workflow guidance motivates explicit evaluation criteria. Our fixture scores are original lab results, not vendor benchmarks.