Skip to main content

Executable inference lab and decision memo

Download the inference lab archive. Extract it and open a terminal in probability-inference. Python 3.10+ and its standard library are sufficient; on Windows replace python3 with py -3.

python3 check.py starter

The initial NotImplementedError is intentional. Implement the four contracts in order: inverse exponential sampling, exponential rate estimation, Beta parameter updating, exact paired randomization. Predict each calibration case by hand before implementing it.

Experiment and reproduction​

python3 run.py starter --seed 110 > results-110.json
python3 run.py starter --seed 221 > results-221.json

Keep both reports. Repeating a seed should reproduce results in the same environment. Do not keep only the report closest to theory. The runner deliberately uses three separate models: synthetic exponential durations, a Bernoulli Bayesian update, and a paired-difference fixture. They are not three analyses of one dataset.

  1. Derive the inverse-CDF formula and demonstrate round-trip recovery of u, including u=1e−20. Explain why log1p matters and why u=1 is rejected.
  2. Compare the synthetic sample mean with 1/2 in units of the theoretical mean standard error 1/(2sqrt(n)). This describes Monte Carlo fluctuation; there is no pass threshold on a lucky seed. Derive the fitted rate and explain the observed survival comparisons.
  3. Compare Beta(1,1) and Beta(2,8) priors after two successes in five trials. Preserve both results and explain the prior sensitivity. Demonstrate that updating disjoint batches matches pooling; duplicate a batch deliberately and explain why the changed answer is unjustified.
  4. Enumerate the randomization distribution of [1,2,3,0] by hand, then check the exact Fraction result. Explain why zero differences create repeated statistic values that still count as separate assignments. The implementation accepts at most 16 integer differences to bound work and make tie comparisons exact.

After an attempt, use python3 check.py reference and compare reference.py. Passing checks establishes agreement on those cases, not the validity of a real sampling design. Fresh extraction and a changed-case demonstration are required for PS12.

Decision memo​

Extend the existing experiment-design memo with a short appendix. All five items are required:

  1. State the synthetic duration model, units, estimated rate, and two ways real latency could violate the uncensored iid exponential assumptions.
  2. Report both seeded outputs and explain sampling error versus model error; do not call in-sample survival agreement external validation.
  3. State the Bayesian prior, likelihood, posterior parameters and next-trial interpretation; distinguish a predictive probability from a credible interval.
  4. State the paired assignment design and sharp null, mean effect, exact p-value, prespecified two-sided rule and a practical effect threshold. Do not infer that the fixture itself establishes random assignment or a real release effect.
  5. Identify what additional data/design would be needed for a release decision, including treatment of timeouts and shared sessions. Explicitly reject posterior interpretations of the p-value and equivalence claims from non-rejection.

The lab and memo share an 8–12 hour allowance within the 25–35 hour extension. Preserve help received, a repaired mistake and an unaided changed case. Reuse the appendix in the semester project; do not invent an extra dataset to fill the template.