Vocabulary
The post’s vocabulary and multivon-eval’s API names, stated once. The API is not being renamed — a rename would break every existing suite for zero information gain.| Post says | multivon-eval says |
|---|---|
| task | EvalCase (a case) |
| trial | one of runs=N (a run) |
| grader | Evaluator |
| transcript | agent trace / conversation transcript |
| outcome | CaseResult status (PASSED, FAILED_QUALITY, JUDGE_ERROR, …) |
The mapping
| Recommendation | What executes it | Since |
|---|---|---|
| Write reference solutions; 0% pass means a broken task, not a bad agent | multivon-eval validate — runs every task’s graders against its reference output, never calls the model; report.zero_pass_cases flags all-trials-failed suspects in every run | 0.16.0 (unreleased — on main) |
| Measure pass@k and pass^k under non-determinism | suite.run(fn, runs=N) + report.pass_at_k(k) / report.pass_hat_k(k) — unbiased estimators, cluster-bootstrap CIs, assert_pass_hat_k gate | 0.16.0 (unreleased — on main) |
| Graduate saturated capability evals | Saturation monitor — report.saturated, report.min_detectable_regression, EvalSuite(purpose="regression") | 0.16.0 (unreleased — on main) |
| Give the judge an “Unknown” way out | UNKNOWN verdicts — hedged judge replies excluded from the score denominator and disclosed, never coerced to yes; the max_error_rate budget keeps judge outages from gating green | 0.16.0 (unreleased — on main) |
| Read the transcripts | multivon-eval view --dir — report browser with per-case judge reasons and side-by-side run diffs | 0.15.0 |
| Use isolated per-dimension judges, not one 1–10 rating | QAG evaluators — binary questions per dimension, auditable down to each verdict | shipped |
| Calibrate the judge against human labels | suite.calibrate() + per-(judge × evaluator) calibrated thresholds with provenance | shipped |
validate), the 100% ceiling (saturation monitor), the single-number middle (pass@k vs pass^k), and the coerced verdict (the judge-UNKNOWN row — hedged judge verdicts parse as UNKNOWN, never as a silent yes).