Reviewing generated proposals against a contract

Use 3–4 hours within the existing 8–12-hour review packet/experiments allocation for this concrete AR1/AR2/AR5 case. It replaces a generic cache-patch rehearsal, not the real-project fitness check, outside review or original ADR requirements. Semester 07 stays 460–655 hours. If prerequisite repair exceeds the allocation, record it separately rather than claiming an established duration.

The proposal is evidence to inspect

A generated explanation can suggest a useful design, but its confidence and plausible examples do not establish correctness. Treat each material claim as a hypothesis. Before reading a proposed patch, write the permitted input, required output, forbidden effect, failure behavior and observability boundary. The same standard applies to human-written patches. Tool provenance belongs in the evidence record; it is not a substitute for a contract.

Consider a tenant-scoped document read. The caller supplies a current access-control set and current document bodies. An allowed request returns the current body; a denied or missing document returns None. Cache lifetime spans calls. Bodies are immutable strings, so None is an unambiguous absence marker. Inputs have valid types. This small sequential contract intentionally excludes concurrent policy changes and remote cache invalidation; record those exclusions before using its results elsewhere.

A proposed optimization says: “return cached content immediately; authorize only a miss.” Its cold-path example works. Warm the cache as Alice, then request the same document as Bob without permission. A hit bypasses the check. Warm it as Alice, revoke Alice, and repeat: the same fault survives even with a tenant-aware key. Correct key identity and current authorization solve different problems.

Exercise: write the state before and after each call. Name the first forbidden observation and the policy clause it violates. Avoid vague findings such as “AI might be insecure.” A precise finding identifies request, state, observable leak and a test that would distinguish a repair from a cosmetic change.

Tests must challenge the proposal's assumptions

A second proposal checks permission first but caches by document ID alone. Alice may legitimately read document 1 in two tenants; the second read can receive the first tenant's body. The authorization check passes while result identity is wrong. A third proposal uses the complete key, rechecks permission and refreshes from the supplied current body. A fourth simply reads the supplied body after authorization without caching. Both can satisfy this bounded contract; the third's cache does not establish a performance improvement.

Download the authored proposal fixture. Write predictions and independent checks before opening check.py; then run python3 check.py. The supplied proposals are teaching fixtures, not collected outputs of a named AI model. The harness checks eight observations per proposal, rejects two defective proposals and a deny-all shortcut, and accepts two valid alternatives. A successful harness run means its expected distinctions held, not that every proposal is safe.

Required probes include cold allowed access, warm unauthorized access, revocation, tenant collision, deletion and changed body. A total assertion like “all denied requests fail” is incomplete unless legitimate reads still work. Tests copied from the patch's explanation may share its assumptions. Derive at least one test from the written contract and one from a boundary the proposal omits. Public examples are calibration, not a secret holdout or population benchmark.

Transfer: a reviewer changes the contract to permit cached body versions for at most 30 seconds while authorization remains current. Identify which tests should change and which must remain. Do not weaken revocation just because body freshness changed. Add clock control before claiming a time-bound result; wall-clock sleeps are not a precise oracle.

Review and repair workflow

1. Freeze the proposed revision and its claim before running checks. Record whether it is authored, generated, edited or observed from a real tool. If using a tool, retain date, model/version as exposed, relevant prompt, selected output and assistance; never invent missing metadata.
2. Write an independent contract and predictions. Do not provide a private reviewer case to the proposal author before freezing the candidate. If it leaks, mark it as practice and use a fresh case; do not call it unseen evidence.
3. Run your discriminating checks. Preserve both accepted and rejected observations and collection failures. A missing test result is unknown, not pass. Do not change the contract merely to accommodate the implementation.
4. Choose accept, repair, reject or defer with a concrete reason. Re-run affected checks after repair and review the changed diff. Keep a valid simpler alternative available; rejection of all automation is not the target outcome.
5. Apply this reasoning to the existing service's actual boundary and obtain the existing AR4 independent review. The fixture does not exercise HTTP routing, distributed revocation or production authorization. Preserve those original obligations.

Calibration within existing gates

AR1 requires separation of proposal claims, observed failures and untested scope. AR2 requires a justified comparison, including the direct-read alternative; no speed claim without measurement. AR5 requires independent probes, valid positive cases, an observed failure, repair, collection-failure handling and the real-project check. Each existing gate must reach level 2; these are required subchecks, not extra averaged points or new gate IDs.

A level-1 response repeats the proposed explanation or runs only the provided harness. A level-2 response derives a fresh counterexample and explains the relevant invariant independently. Level 3 adapts to the changed freshness contract while preserving authorization and identity. After the existing delayed-review interval, use a fresh requirement without the answer key. Real reviewer availability remains a separate requirement.

Sources and later use

NIST SSDF 1.1 supplies a secure-development framework for integrating review and verification into development; this exercise does not claim compliance or certify generated code. Version 1.1 is the identified teaching reference, not a claim to be the newest document forever. Source checked 2026-10-23. Existing architecture and security books remain the selective reading route.

Carry the accepted/rejected proposal and review-cost record into Semester 08's assistance decision, then carry its exact artifact identity into Semester 09's release evidence. Real model use is optional; authored fixtures must remain labeled as such.