Skip to main content
LLMs are non-deterministic. The same input can produce different outputs across runs — especially in agents where variance compounds at every step. A single-run pass/fail tells you very little: did the case fail because your model regressed, or because it got unlucky this time? Multi-run evaluation turns flakiness from an invisible problem into a measurable signal.

​
Run each case multiple times

That’s the only change. Every case now runs 5 times and the results are aggregated:
  • Score: mean across runs
  • Pass rate: fraction of runs that passed
  • Stability: whether the case behaves consistently

​
Reading the results

A case is flaky if it passed at least once but not always. This is the most actionable signal — it means the model is uncertain about that input, not just consistently wrong.

​
Terminal output

The reporter adds pass rate and stability columns automatically when runs > 1:

​
Combine with parallel execution

Run cases in parallel and each case multiple times:
Cases run concurrently; each case’s 5 repetitions run sequentially. Good default for large suites.

​
Statistical significance in experiment comparison

exp.compare() reports an approximate two-proportion test using run summaries. A p-value describes compatibility with the no-difference hypothesis under the sampling assumptions; it is not the probability that a regression is real. A nonsignificant result is not proof of no regression.
For matched cases, save the full reports and use baseline_report.compare(proposal_report). Read the statistical guide for paired tests, missing coverage, multiple comparisons, and limits of the summary-level approximation.

​
CI/CD: fail on instability

More runs = more reliable signal, but proportionally more model calls. Start at runs=3 for most pipelines.

​
How scores are aggregated

For each case across N runs:
  • Score: mean of per-run scores
  • Passed: majority vote — passes if more than half of runs passed
  • Flaky: 0 < pass_count < N (at least one pass and one fail)
  • Latency: mean across runs
Per-evaluator scores in the report also use mean + majority vote, so the evaluator breakdown remains interpretable.

​
Judge reliability

Model flakiness is about your model’s variance. Judge reliability is about the evaluator’s variance — whether the same judge call on the same output produces the same pass/fail decision twice. Enable it once in your config:
The terminal output shows it automatically:
What it measures: In 0.17.0 the SDK re-evaluates eligible LLM graders on a sample of completed (case, output) pairs and bypasses cached judge responses. Deterministic graders and skipped measurements do not inflate agreement. When no eligible measurements exist, the value is None. Agreement measures repeatability, not correctness. A consistently wrong grader can score 100%. Inspect sample size, disagreements, and agreement with independent human labels before gating. There is no universal percentage that establishes fitness for a release decision. See the temperature transport limitation before relying on configured sampling values. Note: reliability_check=True makes additional LLM calls (one re-evaluation pass over reliability_sample cases). Keep reliability_sample low (5–10) for routine runs; increase for audits.