Agent Evaluators

Agent evaluators work with agent_trace — a structured record of what your agent did. They’re framework-agnostic: works with LangChain, LlamaIndex, CrewAI, or any custom agent.

Development protocol: complete judgments

The unreleased agent-judgments/v2 protocol requires a complete Yes/No verdict for every requested criterion. An ambiguous response or provider failure leaves the evaluation unmeasured; it cannot shrink the denominator or become a passing vote. Unexpected grader exceptions remain evaluator errors. Reports retain the attempted prompts, raw replies and verdicts in metadata.evaluation_evidence, including on failed suite evaluations and saved trials. These records can contain the task and tool results: apply your normal report access controls. They do not capture provider requests, usage, hidden SDK retries or process interruption. All seven LLM agent evaluators accept judge=JudgeConfig(...). The three itemwise graders (ToolArgumentAccuracy, ToolCallNecessity, StepFaithfulness) accept max_items=8. A longer trace is skipped before any judgment call; increase the limit deliberately to grade every item. Required-check policies treat missing measurements as indeterminate. A missing trace (None) is not evidence of no tool use. An observed empty trace ([]) has no argument-quality or necessity denominator, so those checks skip. To assert that no tools were called, use ToolCallAccuracy with expected_tool_calls=[] and an observed trace. These LLM scores are heuristics over supplied evidence. For industrial tool workflows, validate tool arguments against the actual schema and verify final state using environment outcome checks. TaskCompletion alone cannot prove that a payment, database update or other external action happened. Tool and memory evidence is passed without silent prefix truncation; provider context limits still apply.

Setting up an agent trace

from multivon_eval import EvalCase, AgentStep, ToolCall

case = EvalCase(
    input="Search for recent AI papers and write a summary",
    agent_trace=[
        AgentStep(
            thought="I need to search for recent papers first",
            tool_calls=[
                ToolCall(
                    name="search",
                    arguments={"query": "AI papers 2025"},
                    result=["Paper A", "Paper B", "Paper C"],
                )
            ],
        ),
        AgentStep(
            thought="Now I'll summarize what I found",
            tool_calls=[ToolCall(name="summarize")],
            output="Here are the key AI papers from 2025...",
        ),
    ],
    expected_tool_calls=["search", "summarize"],
)

ToolCallAccuracy

Checks that the agent called the expected tools. By default, order doesn’t matter (set match). When to use: Regression testing — confirm that new model versions still call the right tools for standard tasks.

from multivon_eval import ToolCallAccuracy

ToolCallAccuracy()                    # unordered set match
ToolCallAccuracy(require_order=True)  # expected tools must appear in order

Score = fraction of expected tools that were called.

`require_order=False`	Score
All expected tools called	1.0
Half expected tools called	0.5
No expected tools called	0.0

When require_order=True, the expected tools are matched as an ordered subsequence. Unrelated calls may appear before, between, or after them; reversed expected calls receive partial credit and still fail at the default threshold. Set penalize_unexpected=True when extra calls should reduce the score. Requires case.agent_trace and case.expected_tool_calls.

Parameter	Type	Default	Description
`require_order`	`bool`	`False`	If `True`, expected tools must appear as an ordered subsequence
`penalize_unexpected`	`bool`	`False`	If `True`, unexpected tool names reduce the score
`threshold`	`float`	`0.7`	Minimum score to pass

ToolArgumentAccuracy

LLM judge that evaluates whether the arguments passed to tools were appropriate and well-formed. When to use: Catching argument-level bugs — wrong field names, missing required params, or semantically incorrect values — that tool call name checks miss.

from multivon_eval import EvalCase, AgentStep, ToolCall, ToolArgumentAccuracy

case = EvalCase(
    input="Search for quarterly reports from 2024",
    agent_trace=[
        AgentStep(tool_calls=[
            ToolCall(name="search", arguments={"query": "quarterly reports 2024", "limit": 10})
        ])
    ],
)
ToolArgumentAccuracy()
ToolArgumentAccuracy(threshold=0.8)

Grades every tool call within max_items (default 8); skips longer traces. Requires a nonempty tool-call trace.

Parameter	Type	Default	Description
`threshold`	`float`	`0.7`	Minimum score to pass

PlanQuality

LLM judge that evaluates the overall quality of the agent’s plan — logic, completeness, and efficiency. When to use: Evaluating the agent’s reasoning process, not just the outcome. Useful when debugging why an agent succeeds or fails on complex multi-step tasks.

from multivon_eval import EvalCase, AgentStep, ToolCall, PlanQuality

case = EvalCase(
    input="Research competitors and draft a comparison table",
    agent_trace=[
        AgentStep(thought="I'll search for each competitor first", tool_calls=[ToolCall(name="search")]),
        AgentStep(thought="Now I'll compile the results", tool_calls=[ToolCall(name="format_table")]),
    ],
)
PlanQuality()
PlanQuality(threshold=0.8)

Assesses:

Does the plan address the task?
Are the steps in a logical order?
Are there unnecessary or redundant steps?
Is anything missing?

Requires case.agent_trace.

Parameter	Type	Default	Description
`threshold`	`float`	`0.7`	Minimum score to pass

TaskCompletion

LLM judge that evaluates whether the agent’s final output actually satisfies the original task. When to use: A judge assessment of whether the supplied output addresses the task. Pair it with deterministic outcome checks when success depends on external state.

from multivon_eval import EvalCase, AgentStep, ToolCall, TaskCompletion

case = EvalCase(
    input="Book a meeting for Monday at 2pm",
    agent_trace=[
        AgentStep(tool_calls=[ToolCall(name="create_event", arguments={"day": "Monday", "time": "14:00"})]),
    ],
)
TaskCompletion()
TaskCompletion(threshold=0.9)

Works with or without agent_trace — if no trace is attached, evaluates the final output alone.

Parameter	Type	Default	Description
`threshold`	`float`	`0.7`	Minimum score to pass

StepFaithfulness

LLM judge that checks whether each step follows logically from the prior steps and the original task. When to use: Catching hallucinated reasoning steps — agents that invent observations, skip over failures, or take actions that contradict earlier tool results.

from multivon_eval import EvalCase, AgentStep, ToolCall, StepFaithfulness

case = EvalCase(
    input="Find the cheapest flight to NYC and book it",
    agent_trace=[
        AgentStep(thought="Searching for flights", tool_calls=[ToolCall(name="search_flights")]),
        AgentStep(thought="Booking the cheapest result found", tool_calls=[ToolCall(name="book_flight")]),
    ],
)
StepFaithfulness()
StepFaithfulness(threshold=0.8)

Grades every step within max_items (default 8); skips longer traces. Requires case.agent_trace.

Parameter	Type	Default	Description
`threshold`	`float`	`0.7`	Minimum score to pass

ToolCallNecessity

Evaluates whether each tool call was actually needed, or if it was redundant given the context and what had already been done.

from multivon_eval import ToolCallNecessity

ToolCallNecessity()

For each tool call, the judge sees all prior calls and asks: was this strictly necessary? Catches agents that over-call tools, re-fetch data they already have, or take “defensive” actions that add no value.

Scoring

score = count(tool calls judged necessary) / count(all tool calls)

Skips if the trace is missing or contains no tool calls. Skips the complete measurement when tool-call count exceeds max_items (default 8). Each tool call is evaluated independently with full prior-call context. The judge prompt is:

“Given what the agent already knows from prior tool calls, was calling {tool_name} with these arguments strictly necessary to make progress on the task?”

Score	Meaning
1.0	All tool calls were necessary
0.5	Half of tool calls were redundant
0.0	Every tool call was unnecessary

Example

# Agent calls search() twice with the same query — second call is redundant
case = EvalCase(
    input="Find the latest Python release",
    agent_trace=[
        AgentStep(tool_calls=[ToolCall(name="search", arguments={"query": "Python latest release"})]),
        AgentStep(tool_calls=[ToolCall(name="search", arguments={"query": "Python latest release"})]),
        AgentStep(output="Python 3.13 was released in October 2024."),
    ],
)
# If the judge marks only the first call necessary, the score is 0.5.

TrajectoryEfficiency

Judge heuristic for redundant steps and recovery. It does not measure an optimal path or compare against an optimal controller.

from multivon_eval import TrajectoryEfficiency

TrajectoryEfficiency()

Scoring

Step 1 — Base score (average of 3 binary QAG questions):

Question	What it catches
Did the agent complete the task without unnecessary detours?	Off-topic steps, tangential tool calls
Is the step count proportionate to task complexity?	Over-engineered solutions
Did the agent avoid repeating steps it had already completed?	Duplicate fetches, re-running the same computation

Each question is answered yes/no by an LLM judge. Base score = mean of the three answers (0.0–1.0). Step 2 — Error recovery penalty: runs when a tool result contains the text error (case insensitive). This heuristic can mistake a successful result for a failure or miss a failure without that word. An ambiguous recovery verdict invalidates the entire measurement.

“Did the agent handle tool failures by retrying with different arguments, switching to an alternative approach, or signalling a clear failure — rather than silently continuing as if the call succeeded?”

If no: score = max(0.0, base_score - 0.2)

final_score ∈ [0.0, 1.0]

Example

# Agent hits a 404 on the first tool call and tries the same call again unchanged
case = EvalCase(
    input="Get the current stock price of AAPL",
    agent_trace=[
        AgentStep(tool_calls=[ToolCall(name="get_price", arguments={"ticker": "AAPL"}, result="Error: 404")]),
        AgentStep(tool_calls=[ToolCall(name="get_price", arguments={"ticker": "AAPL"}, result="Error: 404")]),
        AgentStep(output="I was unable to retrieve the price."),
    ],
)
# Base score might be 0.67 (completes task, proportionate steps, but repeats a failed call)
# Recovery penalty: -0.2 (agent re-ran the same failing call without modification)
# Final TrajectoryEfficiency score: 0.47

AgentMemoryEval

Evaluates whether a multi-session agent uses prior context correctly — retrieving accurately, not hallucinating past context, and forgetting appropriately. When to use: Multi-session assistants, long-running agents, or any system that must carry state across separate conversations. Requires case.context (prior session summary or log) and case.input (current query that needs memory).

from multivon_eval import AgentMemoryEval, EvalCase

case = EvalCase(
    input="What did I ask you to prioritize last session?",
    context="Prior session (2025-11-10): User asked to prioritize the auth module. They mentioned the deadline is end of November.",
    expected_output="auth module",
)
suite.add_evaluators(AgentMemoryEval())
suite.add_evaluators(AgentMemoryEval(threshold=0.8))

Assesses:

Does the response correctly use information from the prior context?
Does it avoid hallucinating facts not in the prior context?
Does it ignore superseded or stale information?
If expected_output is provided, does the response include it?

Parameter	Type	Default	Description
`threshold`	`float`	`0.7`	Minimum score to pass

This grader is a context-use heuristic. It does not implement or reproduce a published memory benchmark.

Full agent eval example

from multivon_eval import (
    EvalSuite, EvalCase, AgentStep, ToolCall,
    ToolCallAccuracy, ToolArgumentAccuracy,
    PlanQuality, TaskCompletion,
)

def run_agent(task: str) -> str:
    # Your agent here — returns final output
    ...

# Build traces from your agent framework
# then wrap in EvalCase
suite = EvalSuite("Agent Eval")
suite.add_cases(cases)
suite.add_evaluators(
    ToolCallAccuracy(require_order=False),
    ToolArgumentAccuracy(),
    ToolCallNecessity(),
    TrajectoryEfficiency(),
    PlanQuality(),
    TaskCompletion(threshold=0.85),
)

report = suite.run(run_agent)

Getting Started

Evaluators

Claude Code Skills

Guides

Reference

Compliance

Development protocol: complete judgments

Setting up an agent trace

ToolCallAccuracy

ToolArgumentAccuracy

PlanQuality

TaskCompletion

StepFaithfulness

ToolCallNecessity

Scoring

Example

TrajectoryEfficiency

Scoring

Example

AgentMemoryEval

Full agent eval example

​Development protocol: complete judgments

​Setting up an agent trace

​ToolCallAccuracy

​ToolArgumentAccuracy

​PlanQuality

​TaskCompletion

​StepFaithfulness

​ToolCallNecessity

​Scoring

​Example

​TrajectoryEfficiency

​Scoring

​Example

​AgentMemoryEval

​Full agent eval example

Development protocol: complete judgments

Setting up an agent trace

ToolCallAccuracy

ToolArgumentAccuracy

PlanQuality

TaskCompletion

StepFaithfulness

ToolCallNecessity

Scoring

Example

TrajectoryEfficiency

Scoring

Example

AgentMemoryEval

Full agent eval example