Skip to main content
New in 0.16.0. A single pass rate hides a distinction that matters in production: a task your agent solves sometimes and a task it solves every time both contribute the same way to pass_rate. Two metrics, computed from the --runs N data you already have, pull them apart — the framing follows Anthropic’s guidance on agent evals: A demo selects for pass@k. A production SLA is pass^k. The gap between them is your flakiness, quantified.

​
The estimators

Both are computed per task from its n recorded trials (c of which passed), then averaged over tasks. pass@k uses the unbiased combinatorial estimator from the HumanEval paper:
pass^k uses the exact hypergeometric analogue:
The naive plug-in (c/n)^k is deliberately not implemented. It samples with replacement from a finite trial pool, which biases the estimate upward — a vanity metric. Concretely, with 3 passes in 5 trials at k=2:

​
From an eval report

No new run mode — the metrics come from the same runs=N data:
Same agent, same data: it can do almost everything (pass@5 = 100% [84%, 100%]) and reliably does almost nothing (pass^5 = 10% [0%, 25%]). The terminal report prints this automatically for any multi-run suite:
Errored and skipped tasks are excluded from the case pool — the same denominator discipline as pass_rate. A judge outage is not a capability signal.

​
Why the CI resamples cases, not trials

The confidence intervals are a cluster bootstrap: tasks are resampled with replacement, the mean of per-task estimators is recomputed, and a percentile interval is taken. Trials within a task are correlated — the same prompt, the same failure modes — so resampling raw trials would treat 20 tasks × 5 trials as 100 independent observations and fake precision the data doesn’t have. The unit of independence is the task; the CI respects that. Degenerate suites (every per-task estimate identical, including all-pass) fall back to a Wilson interval on the mean, so a perfect score still reports ci_low < 1.0 — the same honesty as pass_rate_ci().

​
The honest-UNKNOWN rule

You cannot estimate pass@10 from 5 trials without extrapolating, so multivon-eval doesn’t:
value is None is the contract for UNKNOWN. There is no projected curve and no warning-then-guess. Rerun with runs >= k or lower k.

​
Lottery cases

The tasks driving the pass@k / pass^k gap — passing sometimes, never reliably — are ranked by per-task divergence:
These are your best debugging targets: read the transcripts of a 4/5 task and diff the passing trial against the failing one.

​
Gating on the CI lower bound

assert_pass_hat_k gates on the lower bound of the pass^k CI, not the point estimate — the same EvalGateFailure exit semantics as fail_threshold:
If pass^k is UNKNOWN (k > runs), the gate raises with the UNKNOWN reason rather than silently passing — an ungateable claim must fail loudly.

​
Serialization

When runs > 1, the JSON summary carries both metrics (pass_at_k / pass_hat_k, each with k, value, ci_95, estimator), and view --dir shows a pass^k column. Reports saved before 0.16.0 gain the metrics on load via EvalReport.from_dict — per-case trial counts were already stored, so no migration is needed.