Static multi-turn test scripts assume a fixed conversation path. The moment
your model responds differently, the script is testing a conversation that
never happened. multivon-eval simulate (0.12.0) drives the conversation
live instead: a persona LLM with a profile, a goal, and behavior traits
generates each user turn in response to what your system actually said.
What you get
One result per persona: the full transcript, a stop reason
(goal_reached / max_turns / assistant_refused / budget_exceeded /
driver_error), a goal-completion verdict judged against the persona’s
success_criteria, and scores from the conversation evaluators
(ConversationRelevance, KnowledgeRetention, TurnConsistency) over the
transcript.
Personas
A persona is four fields plus traits:
Author them in JSONL, or let --propose-from PRODUCT.md generate a diverse
set with one LLM call. The proposal prompt always demands at least one
persona with an adversarial trait.
Labels, budgets, and what isn’t claimed
Simulation output is synthetic, and the tool keeps saying so:
- Every result and report carries “simulated personas — measures behavior
under synthetic users, not real traffic.” That label is test-pinned.
- The budget ceiling is hard. The spend estimate prints before the first
call, and hitting
--budget mid-run stops cleanly: completed transcripts
are returned, cut-off personas carry stop_reason="budget_exceeded", and
no partial work is lost to an exception.
- There are no determinism claims. Persona proposal is seeded; conversation
turns are stochastic. The judge model and temperature are recorded in
every result’s metadata so a run is at least describable.
- A
driver_error on one persona never kills the run. It is recorded and
the next persona proceeds.
Simulation with provenance
Each conversation binds its case_uid through the same contextvar the
runtime recorder uses. Run your simulation under
pytest --record-prompts (or wrap it in record_prompts()) and the prompts
your system rendered during each simulated conversation are captured and
bound to that conversation’s case: observed case→site bindings, with the
same propose-only discipline as staleness stamp --from-recordings.
Python API
model_fn has the same contract as EvalSuite.run: one rendered-prompt
string in (the conversation so far, rendered like
EvalCase.conversation_str() as USER: ... / ASSISTANT: ... lines), the
assistant’s reply out.
Because model_fn receives the full conversation, a keyword-routing
test double that matches on the whole string will match its own earlier
replies and derail. Route on the last user message:
prompt.rsplit("USER:", 1)[-1]. (If you ship that bug anyway, the
simulator surfaces it loudly — flat-zero conversation-relevance scores.)
Adversarial personas and refusals
A refusal from your model stops a normal persona’s conversation
(stop_reason="assistant_refused"). Personas with an adversarial trait
are different by design (0.12.1): they exist to probe past refusals, so
a refusal is recorded in the result metadata (refusals_observed) and the
conversation continues to goal or max_turns. One polite “I can’t do
that” at turn 1 is not evidence your system resists a persistent attacker. 
