Generate eval cases

The hardest part of getting started with evals is having nothing to evaluate against. multivon-eval generates cases for you: point it at your documentation and get question-answer pairs in seconds, or expand an existing set with mutations, template grids, and contrast pairs. Two of the generators run with no LLM at all. Every generator stamps provenance (authored_by="generator:<kind>", seed recorded), routes through the dedupe gates, and returns an accounting report: generated N, accepted M — dropped k duplicates, j malformed.

Generate QA cases from docs

generate_from_file() reads a text file and produces question-answer cases grounded in it:

from multivon_eval import generate_from_file, EvalSuite, Faithfulness, Relevance

cases = generate_from_file("docs/faq.md", n=20)

suite = EvalSuite("FAQ Eval")
suite.add_cases(cases)
suite.add_evaluators(Faithfulness(), Relevance())
report = suite.run(my_model)

Parameter	Type	Default	Description
`path`	string	required	Path to the source file (`.txt`, `.md`, `.rst`, `.py`, etc.; read as UTF-8).
`n`	int	`10`	Number of cases to generate.
`task`	string	`"qa"`	One of `"qa"`, `"summarization"`, `"hallucination"`.
`record_spans`	bool	`True`	(task=`qa`) Record each case’s grounding span — `metadata["source_span"]` with start/end offsets and a chunk hash — so a Faithfulness verdict traces to the evidence it came from. Spans that can’t be located are recorded as `None` and counted, not guessed.
`unanswerable_fraction`	float	`0.0`	(task=`qa`) Fraction of cases deliberately not answerable from the text. Their expected behavior is refusal (`expected_output=None`), the cheapest hallucination bait there is.
`return_report`	bool	`False`	Also return the `GenerationReport` with gate accounting. Default keeps the `list[EvalCase]` return shape.

From raw text

generate_from_text() takes the source string directly (same params as generate_from_file plus context_window):

from multivon_eval import generate_from_text

cases = generate_from_text(open("docs/faq.md").read(), n=20, task="qa")

context_window (int, default 3000) caps the characters of source included per generation prompt; long inputs are split into overlapping chunks.

Task types

qa (default) — question/answer pairs with context excerpts. Each case has input (the question), expected_output (the answer), and context (the excerpt). Best for RAG, chatbots, and knowledge-base evaluation.
summarization — document chunks with faithful reference summaries. input is the chunk, expected_output is the summary.
hallucination — faithful-answer cases (expected_output="faithful"). Pair with the Hallucination evaluator to verify your model doesn’t fabricate.

Hallucination benchmark pairs

generate_hallucination_pairs() returns both faithful and hallucinated answer variants — useful for building your own labeled benchmark (HaluEval-style):

from multivon_eval import generate_hallucination_pairs

pairs = generate_hallucination_pairs(my_docs, n=20)
# [{"question", "context", "faithful_answer", "hallucinated_answer"}, ...]

Parameter	Type	Default	Description
`text`	string	required	Source text to ground questions in.
`n`	int	`10`	Number of pairs to generate.

Mutations (free)

CheckList-style robustness suites from cases you already have. Each mutant records its transformation and what you should expect: invariant (the model’s answer shouldn’t materially change) or flip (the old label no longer applies — the expectation is cleared, never silently kept).

from multivon_eval import mutate_cases

mutants, report = mutate_cases(cases, seed=7, per_case=2)
# mutations: typo_noise, whitespace_noise, case_noise,
#            unicode_confusable, punctuation_strip, negation_flip (flip)

multivon-eval generate --mutate cases.jsonl --per-case 2 --seed 7 --output mutants.jsonl

Deterministic per seed. Zero LLM calls.

Template grids (free)

Parametric coverage over axes you define. sample="all" is the full product (capped at 2000); sample="pairwise" is a greedy covering array that hits every value pair at least once.

from multivon_eval import cases_from_template

cases, report = cases_from_template(
    "Refund question about {item} bought {when}",
    axes={"item": ["a laptop", "shoes", "an ebook"],
          "when": ["yesterday", "40 days ago"]},
    sample="pairwise",
)

Rows without expected_output are valid — judge evaluators score outputs without a reference answer. No label is ever invented.

Contrast pairs

For each passing case, an LLM proposes a minimally-edited unfaithful twin, and a judge verifies the flip is real before the twin is accepted. Rejected twins are counted in the report, not kept. Twins share a pair_id with their source case, so downstream comparisons are genuinely paired.

from multivon_eval import generate_contrast_pairs

pairs, report = generate_contrast_pairs(cases, verify=True, budget_usd=1.0)
# Needs cases that carry both `context` and `expected_output` — a twin is a
# minimally-edited UNFAITHFUL answer against the same context. Cases without
# context are skipped (and counted in the report).

Simulation transcripts as datasets

multivon-eval simulate ... --export-cases cases.jsonl turns persona conversation transcripts into conversation-shaped EvalCases (empty transcripts are skipped and counted). See the simulate guide.

CLI

# QA generation from a file or dir
multivon-eval generate --from docs/faq.md --n 20 --task qa --output cases.jsonl

# Mutations, template grids, contrast pairs
multivon-eval generate --mutate cases.jsonl --per-case 2 --seed 7

Flag	Description
`--from <path>`	Source file.
`--text <text>`	Raw text source (alternative to `--from`).
`--n <int>`	Number of cases. Defaults to `10`.
`--task`	One of `qa`, `summarization`, `hallucination`. Defaults to `qa`.
`--output`, `-o`	Save to JSONL. If omitted, prints a preview to stdout.
`--mutate`	Deterministic robustness mutations (also `--per-case`, `--seed`).
`--template` / `--axes` / `--sample`	Template-grid expansion.
`--contrast` / `--no-verify`	Contrast-pair generation.

Tips

Start small — generate 10-20 cases first, review them, then scale up.
Use your actual docs — cases from your real content catch real problems.
Mix with manual cases — generated cases cover breadth; manual cases cover the edge cases you already know about.
Task choice matters — qa for RAG, summarization for summarization pipelines, hallucination to stress-test faithfulness.

Getting Started

Evaluators

Claude Code Skills

Guides

Reference

Compliance

Generate QA cases from docs

From raw text

Task types

Hallucination benchmark pairs

Mutations (free)

Template grids (free)

Contrast pairs

Simulation transcripts as datasets

CLI

Tips

​Generate QA cases from docs

​From raw text

​Task types

​Hallucination benchmark pairs

​Mutations (free)

​Template grids (free)

​Contrast pairs

​Simulation transcripts as datasets

​CLI

​Tips

Generate QA cases from docs

From raw text

Task types

Hallucination benchmark pairs

Mutations (free)

Template grids (free)

Contrast pairs

Simulation transcripts as datasets

CLI

Tips