Skip to main content
LLMs are non-deterministic. The same input can produce different outputs across runs — especially in agents where variance compounds at every step. A single-run pass/fail tells you very little: did the case fail because your model regressed, or because it got unlucky this time? Multi-run evaluation turns flakiness from an invisible problem into a measurable signal.

​
Run each case multiple times

That’s the only change. Every case now runs 5 times and the results are aggregated:
  • Score: mean across runs
  • Pass rate: fraction of runs that passed
  • Stability: whether the case behaves consistently

​
Reading the results

A case is flaky if it passed at least once but not always. This is the most actionable signal — it means the model is uncertain about that input, not just consistently wrong.

​
Terminal output

The reporter adds pass rate and stability columns automatically when runs > 1:

​
Combine with parallel execution

Run cases in parallel and each case multiple times:
Cases run concurrently; each case’s 5 repetitions run sequentially. Good default for large suites.

​
Statistical significance in experiment comparison

When comparing two runs, exp.compare() now shows whether the difference is real or sampling noise:
vs a smaller dataset:
Same delta, different conclusions — because with 10 cases, a 7% change is within noise. With 100 cases, it’s real. Significance levels:
  • p<0.01 ✦✦ — highly significant, very unlikely to be noise
  • p<0.05 ✦ — significant at the standard threshold
  • p<0.10 — marginal, treat with caution
  • p≥0.10 — not significant, likely sampling noise

​
CI/CD: fail on instability

More runs = more reliable signal, but proportionally more model calls. Start at runs=3 for most pipelines.

​
How scores are aggregated

For each case across N runs:
  • Score: mean of per-run scores
  • Passed: majority vote — passes if more than half of runs passed
  • Flaky: 0 < pass_count < N (at least one pass and one fail)
  • Latency: mean across runs
Per-evaluator scores in the report also use mean + majority vote, so the evaluator breakdown remains interpretable.

​
Judge reliability

Model flakiness is about your model’s variance. Judge reliability is about the evaluator’s variance — whether the same judge call on the same output produces the same pass/fail decision twice. Enable it once in your config:
The terminal output shows it automatically:
What it measures: After the main eval, the SDK re-runs all evaluators on a random sample of (case, output) pairs and measures how often the judge gives the same pass/fail decision. Low agreement means your eval scores contain noise from the judge, not just from your model. Thresholds:
  • ≥ 85%: reliable for CI gating
  • 70–85%: usable for iteration; add more cases to average out judge variance
  • < 70%: judge is significantly non-deterministic — lower temperature, use a larger judge model, or pin questions= in CheckEvaluator
Note: reliability_check=True makes additional LLM calls (one re-evaluation pass over reliability_sample cases). Keep reliability_sample low (5–10) for routine runs; increase for audits.