AI Engineering Examination

Time: four hours plus a 25-minute oral defense. Resources: official documentation only.

Part A: Mechanisms

Trace backpropagation, masked attention, lexical and dense retrieval, reranking, calibration, and queueing. State shapes, assumptions, and failure modes.

Part B: Evaluation

Given predictions, relevance judgments, subgroup labels, latency, and cost data, calculate appropriate metrics, uncertainty, threshold, ranking quality, and cost per successful outcome. Identify leakage and evaluator bias.

Part C: Secure design

Threat-model an AI agent with retrieval and two tools. Specify authorization outside the model, prompt-injection tests, privacy controls, audit evidence, safe failure, and residual risk.

Part D: Operations

Diagnose a canary whose mean quality is stable while tail latency, one language slice, and escalation cost regress. Issue a ship, limit, rollback, or stop decision and write the first six runbook actions.

A correct answer states what evidence is missing. Unsupported certainty loses credit.