Evaluate retained OpenTelemetry traces

Experimental in PyPI 0.19.0. Install with pip install 'multivon-eval[otel]==0.19.0'. Instrumentation, collection, transport, storage, and backend access stay with OpenTelemetry and your existing tools. The bridge grades a captured text/tool execution without running the target again. It retains original OTLP bytes alongside the projected case, so resource metadata, schema URLs, errors, usage attributes and unsupported fields survive. Your suite’s LLM graders can still make judge calls; use deterministic graders for an entirely offline analysis.

Import a bounded execution

Capture native, uncompressed ExportTraceServiceRequest bodies through your existing exporter or Collector. Supply the trace ID and the span that bounds the task. Do not pass compressed HTTP bodies, a whole JSONL file, or generic protobuf JSON with base64 IDs as one request.

from pathlib import Path
from multivon_eval import EvalCase, EvalSuite, ExactMatch, ToolCallAccuracy
from multivon_eval.integrations.otel import GENAI_PROFILE, OtelTrace, score_otel_trace

# trace_id/root_span_id come from the captured application execution.
trace = OtelTrace(
    payloads=(Path("trace-request.pb").read_bytes(),),
    trace_id=trace_id,
    root_span_id=root_span_id,
    profile=GENAI_PROFILE,
    tool_coverage_complete=False,
)
case = EvalCase("Look up order 7", "shipped", case_id="order-7",
                source_id="order-7-source", expected_tool_calls=["lookup_order"])
suite = EvalSuite("order workflow").add_evaluators(ExactMatch(), ToolCallAccuracy())
report = score_otel_trace(suite, trace, case)
report.save_json("order-report.json")

For the Collector file exporter’s newline-delimited OTLP JSON, use payloads=tuple(Path("traces.jsonl").read_bytes().splitlines()) and encoding="json". Standard OTLP JSON uses hexadecimal trace/span IDs and integer enum values. The bridge adapts these IDs for the upstream protobuf JSON parser; it preserves the original bytes, including unknown fields. The Collector file exporter is alpha upstream: pin its version and verify your export format. The root’s last user message must match case.input. Output comes from the root unless you select a descendant with output_span_id=. Input/output use gen_ai.input.messages / gen_ai.output.messages with text parts. Multiple assistant candidates and media parts require a richer evaluation path; this adapter cannot silently flatten them into one answer. Missing/redacted output is unmeasured, and a selected operation’s native error remains an execution error.

Understand capture completeness

tool_coverage_complete=True is an explicit assertion by the caller about instrumentation and collection for this execution. Use it only when you have established that boundary. A sampling flag or an empty span list cannot prove that no uninstrumented tool ran. The example leaves this assertion false: tool checks remain unmeasured and an acceptance policy returns indeterminate. Tool projection recognizes gen_ai.operation.name="execute_tool" and MCP mcp.method.name="tools/call". It needs a tool name, object arguments and a captured result. Instrumentation must opt into capturing this content. Treat argument/result retention according to your application’s data policy. Dropped data, missing parents, invalid timestamps, unsupported status, tool errors and overlapping/nested tool spans become evidence issues. The adapter does not guess a total order, merge client/server spans into an assumed single call, or replace absent arguments with {}. Exact transport retries are deduplicated by span identity; conflicting duplicates are rejected. The saved trial keeps the original case identity separate from its observed trace. Native request batches may include other traces: their bytes also remain in the saved evidence. Select and retain batches with that scope in mind. Hashes detect changes relative to retained artifacts; they do not authenticate instrumentation, the caller’s assertion, or externally changed state.

Export evaluation events through your logger

from multivon_eval.integrations.otel_export import emit_evaluation_events

# logger_provider is your configured OpenTelemetry SDK LoggerProvider.
submitted = emit_evaluation_events(report, logger_provider.get_logger("my.evals"))
flushed = logger_provider.force_flush()

Each saved grader emits a standard gen_ai.evaluation.result log event. Measured events include a score and pass/fail label; skipped/error measurements have an unmeasured label and no invented numeric score. A separate multivon.evaluation.trial event preserves trial status even when no grader ran. Imported events link to the evaluated native trace and selected output span. Other reports emit unparented events instead of attaching an unrelated ambient request. Explanations are opt-in with include_explanations=True. The function validates retained trial integrity before submitting events and does not install global providers. The returned count means submitted to the logger, not delivered to a backend. Your processors/exporters own retries, limits, delivery and shutdown. Native usage attributes remain available in the retained trace; this bridge does not yet reconcile provider billing or capture judge requests automatically.

Reproduce the interoperability check

python examples/otel_evidence.py --output-dir otel-demo

This offline fixture uses the official Python SDK and OTLP HTTP exporters for trace and event round trips. To exercise published multivon-mcp over real stdio, install the MCP client SDK in the example environment and pass --mcp-python /path/to/python-with-multivon-mcp. It checks both complete and incomplete reports through eval_acceptance_report, without model calls. See the validation record for versions, the actual Collector fixture, and compatibility limits. The profile pins the upstream GenAI conventions revision c88d504. Those conventions are Development, not stable. SDK version, convention revision and OTLP schema URLs are separate identifiers. Unsupported profiles are rejected; schema URLs are retained without automatic migration. The OTLP JSON specification and Collector file exporter define the upstream formats used here.

Getting Started

Evaluators

Claude Code Skills

Guides

Reference

Compliance

Evaluate retained OpenTelemetry traces

Import a bounded execution

Understand capture completeness

Export evaluation events through your logger

Reproduce the interoperability check

​Import a bounded execution

​Understand capture completeness

​Export evaluation events through your logger

​Reproduce the interoperability check

Import a bounded execution

Understand capture completeness

Export evaluation events through your logger

Reproduce the interoperability check