> ## Documentation Index
> Fetch the complete documentation index at: https://docs.multivon.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Evaluators

> Evaluate tool use, planning, and task completion in agentic systems.

Agent evaluators work with `agent_trace` — a structured record of what your agent did. They're framework-agnostic: works with LangChain, LlamaIndex, CrewAI, or any custom agent.

## Setting up an agent trace

```python theme={null}
from multivon_eval import EvalCase, AgentStep, ToolCall

case = EvalCase(
    input="Search for recent AI papers and write a summary",
    agent_trace=[
        AgentStep(
            thought="I need to search for recent papers first",
            tool_calls=[
                ToolCall(
                    name="search",
                    arguments={"query": "AI papers 2025"},
                    result=["Paper A", "Paper B", "Paper C"],
                )
            ],
        ),
        AgentStep(
            thought="Now I'll summarize what I found",
            tool_calls=[ToolCall(name="summarize")],
            output="Here are the key AI papers from 2025...",
        ),
    ],
    expected_tool_calls=["search", "summarize"],
)
```

## ToolCallAccuracy

Checks that the agent called the expected tools. By default, order doesn't matter (set match).

**When to use:** Regression testing — confirm that new model versions still call the right tools for standard tasks.

```python theme={null}
from multivon_eval import ToolCallAccuracy

ToolCallAccuracy()                    # unordered set match
ToolCallAccuracy(require_order=True)  # must match in exact order
```

Score = fraction of expected tools that were called.

| `require_order=False`      | Score |
| -------------------------- | ----- |
| All expected tools called  | 1.0   |
| Half expected tools called | 0.5   |
| No expected tools called   | 0.0   |

When `require_order=True`, uses sequence alignment — partially correct order scores between 0 and 1.

Requires `case.agent_trace` and `case.expected_tool_calls`.

| Parameter       | Type    | Default | Description                                               |
| --------------- | ------- | ------- | --------------------------------------------------------- |
| `require_order` | `bool`  | `False` | If `True`, tools must be called in the exact listed order |
| `threshold`     | `float` | `0.7`   | Minimum score to pass                                     |

## ToolArgumentAccuracy

LLM judge that evaluates whether the arguments passed to tools were appropriate and well-formed.

**When to use:** Catching argument-level bugs — wrong field names, missing required params, or semantically incorrect values — that tool call name checks miss.

```python theme={null}
from multivon_eval import EvalCase, AgentStep, ToolCall, ToolArgumentAccuracy

case = EvalCase(
    input="Search for quarterly reports from 2024",
    agent_trace=[
        AgentStep(tool_calls=[
            ToolCall(name="search", arguments={"query": "quarterly reports 2024", "limit": 10})
        ])
    ],
)
ToolArgumentAccuracy()
ToolArgumentAccuracy(threshold=0.8)
```

Evaluates up to 8 tool calls. Requires `case.agent_trace`.

| Parameter   | Type    | Default | Description           |
| ----------- | ------- | ------- | --------------------- |
| `threshold` | `float` | `0.7`   | Minimum score to pass |

## PlanQuality

LLM judge that evaluates the overall quality of the agent's plan — logic, completeness, and efficiency.

**When to use:** Evaluating the agent's reasoning process, not just the outcome. Useful when debugging why an agent succeeds or fails on complex multi-step tasks.

```python theme={null}
from multivon_eval import EvalCase, AgentStep, ToolCall, PlanQuality

case = EvalCase(
    input="Research competitors and draft a comparison table",
    agent_trace=[
        AgentStep(thought="I'll search for each competitor first", tool_calls=[ToolCall(name="search")]),
        AgentStep(thought="Now I'll compile the results", tool_calls=[ToolCall(name="format_table")]),
    ],
)
PlanQuality()
PlanQuality(threshold=0.8)
```

Assesses:

* Does the plan address the task?
* Are the steps in a logical order?
* Are there unnecessary or redundant steps?
* Is anything missing?

Requires `case.agent_trace`.

| Parameter   | Type    | Default | Description           |
| ----------- | ------- | ------- | --------------------- |
| `threshold` | `float` | `0.7`   | Minimum score to pass |

## TaskCompletion

LLM judge that evaluates whether the agent's final output actually satisfies the original task.

**When to use:** End-to-end success metric — the primary evaluator for whether the agent delivered. Use alongside trajectory evaluators to distinguish "right answer, bad process" from "wrong answer".

```python theme={null}
from multivon_eval import EvalCase, AgentStep, ToolCall, TaskCompletion

case = EvalCase(
    input="Book a meeting for Monday at 2pm",
    agent_trace=[
        AgentStep(tool_calls=[ToolCall(name="create_event", arguments={"day": "Monday", "time": "14:00"})]),
    ],
)
TaskCompletion()
TaskCompletion(threshold=0.9)
```

Works with or without `agent_trace` — if no trace is attached, evaluates the final output alone.

| Parameter   | Type    | Default | Description           |
| ----------- | ------- | ------- | --------------------- |
| `threshold` | `float` | `0.7`   | Minimum score to pass |

## StepFaithfulness

LLM judge that checks whether each step follows logically from the prior steps and the original task.

**When to use:** Catching hallucinated reasoning steps — agents that invent observations, skip over failures, or take actions that contradict earlier tool results.

```python theme={null}
from multivon_eval import EvalCase, AgentStep, ToolCall, StepFaithfulness

case = EvalCase(
    input="Find the cheapest flight to NYC and book it",
    agent_trace=[
        AgentStep(thought="Searching for flights", tool_calls=[ToolCall(name="search_flights")]),
        AgentStep(thought="Booking the cheapest result found", tool_calls=[ToolCall(name="book_flight")]),
    ],
)
StepFaithfulness()
StepFaithfulness(threshold=0.8)
```

Evaluates up to 8 steps. Requires `case.agent_trace`.

| Parameter   | Type    | Default | Description           |
| ----------- | ------- | ------- | --------------------- |
| `threshold` | `float` | `0.7`   | Minimum score to pass |

***

## ToolCallNecessity

Evaluates whether each tool call was actually needed, or if it was redundant given the context and what had already been done.

```python theme={null}
from multivon_eval import ToolCallNecessity

ToolCallNecessity()
```

For each tool call, the judge sees all prior calls and asks: was this strictly necessary? Catches agents that over-call tools, re-fetch data they already have, or take "defensive" actions that add no value.

### Scoring

```
score = count(tool calls judged necessary) / count(all tool calls)
```

Returns `1.0` if the agent made no tool calls. Capped at 8 tool calls per trace to control cost.

Each tool call is evaluated independently with full prior-call context. The judge prompt is:

> "Given what the agent already knows from prior tool calls, was calling `{tool_name}` with these arguments strictly necessary to make progress on the task?"

| Score | Meaning                           |
| ----- | --------------------------------- |
| 1.0   | All tool calls were necessary     |
| 0.5   | Half of tool calls were redundant |
| 0.0   | Every tool call was unnecessary   |

### Example

```python theme={null}
# Agent calls search() twice with the same query — second call is redundant
case = EvalCase(
    input="Find the latest Python release",
    agent_trace=[
        AgentStep(tool_calls=[ToolCall(name="search", arguments={"query": "Python latest release"})]),
        AgentStep(tool_calls=[ToolCall(name="search", arguments={"query": "Python latest release"})]),
        AgentStep(output="Python 3.13 was released in October 2024."),
    ],
)
# Expected ToolCallNecessity score: 0.5 (1 of 2 calls necessary)
```

## TrajectoryEfficiency

Evaluates whether the agent took the most efficient path to the answer, and whether it recovered correctly from tool failures.

```python theme={null}
from multivon_eval import TrajectoryEfficiency

TrajectoryEfficiency()
```

### Scoring

**Step 1 — Base score** (average of 3 binary QAG questions):

| Question                                                      | What it catches                                    |
| ------------------------------------------------------------- | -------------------------------------------------- |
| Did the agent complete the task without unnecessary detours?  | Off-topic steps, tangential tool calls             |
| Is the step count proportionate to task complexity?           | Over-engineered solutions                          |
| Did the agent avoid repeating steps it had already completed? | Duplicate fetches, re-running the same computation |

Each question is answered yes/no by an LLM judge. Base score = mean of the three answers (0.0–1.0).

**Step 2 — Error recovery penalty** (runs only when failed tool calls are detected):

> "Did the agent handle tool failures by retrying with different arguments, switching to an alternative approach, or signalling a clear failure — rather than silently continuing as if the call succeeded?"

If **no**: `score = max(0.0, base_score - 0.2)`

```
final_score ∈ [0.0, 1.0]
```

### Example

```python theme={null}
# Agent hits a 404 on the first tool call and tries the same call again unchanged
case = EvalCase(
    input="Get the current stock price of AAPL",
    agent_trace=[
        AgentStep(tool_calls=[ToolCall(name="get_price", arguments={"ticker": "AAPL"}, result="Error: 404")]),
        AgentStep(tool_calls=[ToolCall(name="get_price", arguments={"ticker": "AAPL"}, result="Error: 404")]),
        AgentStep(output="I was unable to retrieve the price."),
    ],
)
# Base score might be 0.67 (completes task, proportionate steps, but repeats a failed call)
# Recovery penalty: -0.2 (agent re-ran the same failing call without modification)
# Final TrajectoryEfficiency score: 0.47
```

## AgentMemoryEval

Evaluates whether a multi-session agent uses prior context correctly — retrieving accurately, not hallucinating past context, and forgetting appropriately.

**When to use:** Multi-session assistants, long-running agents, or any system that must carry state across separate conversations.

Requires `case.context` (prior session summary or log) and `case.input` (current query that needs memory).

```python theme={null}
from multivon_eval import AgentMemoryEval, EvalCase

case = EvalCase(
    input="What did I ask you to prioritize last session?",
    context="Prior session (2025-11-10): User asked to prioritize the auth module. They mentioned the deadline is end of November.",
    expected_output="auth module",
)
suite.add_evaluators(AgentMemoryEval())
suite.add_evaluators(AgentMemoryEval(threshold=0.8))
```

Assesses:

* Does the response correctly use information from the prior context?
* Does it avoid hallucinating facts not in the prior context?
* Does it ignore superseded or stale information?
* If `expected_output` is provided, does the response include it?

| Parameter   | Type    | Default | Description           |
| ----------- | ------- | ------- | --------------------- |
| `threshold` | `float` | `0.7`   | Minimum score to pass |

Aligned with [AMA-Bench (2026)](https://arxiv.org/abs/2602.22769), a benchmark for evaluating long-horizon memory in agentic applications.

## Full agent eval example

```python theme={null}
from multivon_eval import (
    EvalSuite, EvalCase, AgentStep, ToolCall,
    ToolCallAccuracy, ToolArgumentAccuracy,
    PlanQuality, TaskCompletion,
)

def run_agent(task: str) -> str:
    # Your agent here — returns final output
    ...

# Build traces from your agent framework
# then wrap in EvalCase
suite = EvalSuite("Agent Eval")
suite.add_cases(cases)
suite.add_evaluators(
    ToolCallAccuracy(require_order=False),
    ToolArgumentAccuracy(),
    ToolCallNecessity(),
    TrajectoryEfficiency(),
    PlanQuality(),
    TaskCompletion(threshold=0.85),
)

report = suite.run(run_agent)
```