Statistical Rigor

Pass Rate: 80% [69%–89% 95% CI]   Avg Score: 0.82 [0.74–0.90]
Score distribution  p10:0.41  p50:0.88  p90:0.96

lo, hi = report.pass_rate_ci()        # Wilson 95% CI on pass rate
lo, hi = report.avg_score_ci()        # bootstrap 95% CI on mean score
pct    = report.score_percentiles()   # {"p10": 0.41, "p50": 0.88, "p90": 0.96}

from multivon_eval import wilson_interval

# 80 passing out of 100 cases
lo, hi = wilson_interval(80, 100)
print(f"95% CI: [{lo:.1%}, {hi:.1%}]")
# → 95% CI: [71.1%, 86.7%]

# Small test suite: 8 of 10 passing
lo, hi = wilson_interval(8, 10)
print(f"95% CI: [{lo:.1%}, {hi:.1%}]")
# → 95% CI: [49.0%, 94.3%]
# Wide interval — you can't conclude much from 10 cases

  95% CI (before): [71.4%, 89.3%]
  95% CI (after):  [83.5%, 96.2%]
  Statistical significance: p=0.01 ✦✦ highly significant
  Verdict: IMPROVED — pass rate up +12.0%

from multivon_eval import runs_needed

# How many test cases to detect a 10% improvement (80% power)?
n = runs_needed(delta=0.10)
# → 291

# Smaller improvement requires more cases
n = runs_needed(delta=0.05)
# → 1248

# Higher power threshold
n = runs_needed(delta=0.10, power=0.90)
# → 389

# Different baseline pass rate
n = runs_needed(delta=0.10, baseline=0.85)
# → 138

  Statistical significance: p=0.23 not significant (likely noise)
  Hint: need ≥291 test cases to detect this 10% delta at 80% power.
  Verdict: NO MEANINGFUL CHANGE (delta +10.0%, not significant)

report.saturated                    # True when every EVALUATED task passed
report.min_detectable_regression    # smallest pass-rate DROP detectable at 80% power

  ⚠ Saturated: 40/40 trials passed. All this run can claim is a pass rate
  ≥ 91.2% (95% Wilson). At n=40, the smallest regression this suite can
  detect at 80% power is ~14% — a real 5pp quality drop would look like
  noise. Graduate this suite to a regression suite (purpose='regression')
  and add harder capability tasks.

suite = EvalSuite("smoke-tests", purpose="capability")   # saturation = nag to graduate
suite = EvalSuite("smoke-tests", purpose="regression")   # saturation = expected; warning inverts

report = suite.run(model_fn, runs=10)

print(f"Stability: {report.stability_score:.0%}")   # % of non-flaky cases
print(f"Flaky cases: {report.flaky_count}")

for cr in report.case_results:
    lo, hi = wilson_interval(cr.pass_count, cr.runs)
    print(f"  {cr.case_input[:40]}: {cr.run_pass_rate:.0%} [{lo:.0%}, {hi:.0%}]")

  Evaluator scores         Before           After    BH-adj p
  ────────────────────────────────────────────────────────────
  faithfulness             0.7800  →   0.6800  ↓   0.023 *
  context_precision        0.9200  →   0.7900  ↓   0.034 *
  relevance                0.8800  →   0.8600        0.412

  (* significant after Benjamini-Hochberg correction, FDR 5%)

from multivon_eval import benjamini_hochberg

# p-values from 5 simultaneous tests
raw = [0.001, 0.040, 0.030, 0.200, 0.800]
adj = benjamini_hochberg(raw)
# → [0.005, 0.067, 0.067, 0.250, 0.800]
# Only the first test survives at FDR 5%

result = suite.calibrate([
    (EvalCase(input="How do I cancel?"), "Please contact billing.", False),
    (EvalCase(input="Reset password?"),  "Click Forgot Password.",  True),
    # ... more (case, output, human_pass) tuples
])
print(result)

Judge Calibration — 50 labeled cases
  Agreement:  88.0%
  Precision:  84.0%
  Recall:     91.0%
  F1 Score:   87.4%
  By evaluator:
    faithfulness: agreement=90.0%  F1=89.0%
    relevance:    agreement=82.0%  F1=80.0%