> ## Documentation Index
> Fetch the complete documentation index at: https://docs.multivon.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Simulate — persona-driven multi-turn evaluation

> Drive adaptive conversations against your live system with synthetic personas, scored by the conversation evaluators. Budgets and provenance built in.

Static multi-turn test scripts assume a fixed conversation path. The moment
your model responds differently, the script is testing a conversation that
never happened. `multivon-eval simulate` (0.12.0) drives the conversation
live instead: a persona LLM with a profile, a goal, and behavior traits
generates each user turn in response to what your system actually said.

```bash theme={null}
multivon-eval simulate \
    --model-cmd model.py \          # exposes model_fn(prompt) -> str
    --personas personas.jsonl \     # or: --propose-from PRODUCT.md --n-personas 5
    --max-turns 8 \
    --budget 1.00 \
    --out results.jsonl
```

## What you get

One result per persona: the full transcript, a stop reason
(`goal_reached` / `max_turns` / `assistant_refused` / `budget_exceeded` /
`driver_error`), a goal-completion verdict judged against the persona's
`success_criteria`, and scores from the conversation evaluators
(`ConversationRelevance`, `KnowledgeRetention`, `TurnConsistency`) over the
transcript.

## Personas

A persona is four fields plus traits:

```json theme={null}
{"name": "rushed_customer",
 "profile": "A customer in a hurry who wants a refund for order #1234.",
 "goal": "Find out how to get a refund and confirm the refund window.",
 "success_criteria": "The assistant stated the refund window and the process.",
 "traits": ["terse", "impatient"]}
```

Author them in JSONL, or let `--propose-from PRODUCT.md` generate a diverse
set with one LLM call. The proposal prompt always demands at least one
persona with an `adversarial` trait.

## Labels, budgets, and what isn't claimed

Simulation output is synthetic, and the tool keeps saying so:

* Every result and report carries **"simulated personas — measures behavior
  under synthetic users, not real traffic."** That label is test-pinned.
* The budget ceiling is hard. The spend estimate prints before the first
  call, and hitting `--budget` mid-run stops cleanly: completed transcripts
  are returned, cut-off personas carry `stop_reason="budget_exceeded"`, and
  no partial work is lost to an exception.
* There are no determinism claims. Persona proposal is seeded; conversation
  turns are stochastic. The judge model and temperature are recorded in
  every result's metadata so a run is at least *describable*.
* A `driver_error` on one persona never kills the run. It is recorded and
  the next persona proceeds.

## Simulation with provenance

Each conversation binds its `case_uid` through the same contextvar the
[runtime recorder](/guides/staleness#runtime-recordings) uses. Run your simulation under
`pytest --record-prompts` (or wrap it in `record_prompts()`) and the prompts
your system rendered during each simulated conversation are captured and
bound to that conversation's case: observed case→site bindings, with the
same propose-only discipline as `staleness stamp --from-recordings`.

## Python API

```python theme={null}
from multivon_eval import simulate, Persona, score_simulations

results = simulate(model_fn, personas, max_turns=8, budget_usd=1.00)
summary = score_simulations(results)
```

`model_fn` has the same contract as `EvalSuite.run`: one rendered-prompt
string in (the conversation so far, rendered like
`EvalCase.conversation_str()` as `USER: ...` / `ASSISTANT: ...` lines), the
assistant's reply out.

<Warning>
  Because `model_fn` receives the **full** conversation, a keyword-routing
  test double that matches on the whole string will match its *own earlier
  replies* and derail. Route on the last user message:
  `prompt.rsplit("USER:", 1)[-1]`. (If you ship that bug anyway, the
  simulator surfaces it loudly — flat-zero conversation-relevance scores.)
</Warning>

## Adversarial personas and refusals

A refusal from your model stops a normal persona's conversation
(`stop_reason="assistant_refused"`). Personas with an `adversarial` trait
are different by design (0.12.1): they exist to probe *past* refusals, so
a refusal is recorded in the result metadata (`refusals_observed`) and the
conversation continues to goal or `max_turns`. One polite "I can't do
that" at turn 1 is not evidence your system resists a persistent attacker.