Experimental world-model evaluation

This experimental interface is available in PyPI 0.19.0. The initial profile evaluates fully observed numeric state vectors. Video, latent-state decoding, hidden-object memory and real-robot control are outside its tested scope. Agent reasoning traces use different evaluators.

Measure the capability you need

Question	Evidence to collect
Does the model respond to actions?	Matched initial states, different actions, predicted versus observed effects
Does prediction deteriorate with horizon?	Per-coordinate errors at specified steps, with source counts and missing endpoints
Does state persist appropriately?	Controlled initial-state changes and their observed later effects
Is uncertainty useful?	Empirical interval coverage and width, with the declared distribution assumptions
Does the model help decisions?	The same planner/budget operating in fresh real environment instances

These are established evaluation concerns, not new Multivon metrics. See the decision-focused position paper. Use upstream video tools such as WorldFoundry or WorldModelBench for their supported outputs; their results do not establish this profile’s control performance.

Capture an actual reference and forecast

Install the existing Gymnasium extra:

pip install -e '.[gymnasium]'

import gymnasium as gym

from multivon_eval import EvalCase
from multivon_eval.dynamics import capture_forecast
from multivon_eval.dynamics_metrics import forecast_metrics
from multivon_eval.integrations.gymnasium import capture_episode

actions = [0, 1, 0, 1]  # fix the complete plan before executing the reference
holder = {}

def factory():
    holder["env"] = gym.make("CartPole-v1")
    return holder["env"]

def interact(env, observation, info):
    for action in actions:
        _, _, terminated, truncated, _ = env.step(action)
        if terminated or truncated:
            break

reference = capture_episode(
    factory, interact,
    case=EvalCase("Predict these CartPole actions", case_id="cartpole:7", source_id="seed:7"),
    environment_id="CartPole-v1/native", observer_id="native-state/v1",
    observe=lambda: {"state": list(holder["env"].unwrapped.state)},
    seed=7, max_steps=len(actions),
)

def persistence(request):
    return {"states": [list(request["initial_state"]) for _ in request["actions"]]}

forecast = capture_forecast(
    reference, persistence, model_id="persistence/v1", contract="cartpole-vector/v1",
    coordinates={"x": "m", "velocity": "m/s", "angle": "rad", "angular_velocity": "rad/s"},
    horizons=[1, 4], planned_actions=actions,
)
print(forecast_metrics(forecast))

The model receives a detached object containing only initial_state and actions. It returns exactly one vector per supplied action. It may also return aligned standard_deviation vectors for marginal Gaussian diagnostics and a portable metadata object describing the method. Positive finite deviations, finite coordinates and matching dimensions are required. Do not attach future truth through the callback’s closure or global state. planned_actions must match the actually executed prefix. Freeze that sequence before simulation: its length must not disclose when the reference terminates. If omitted, the API explicitly records action_scope="observed_prefix"; the model then knows the reference-dependent sequence length. This mode is useful for saved-prefix diagnostics but cannot claim a predeclared forecast experiment.

Preserve missing evidence

ForecastEvidence retains the bound native episode, request, returned forecast, model/contract IDs and errors. References with simulator/setup/cleanup failures receive simulator_error, and the model is not called. Failed or malformed model responses receive model_error. Neither becomes a quality score. A digest checks content consistency, not authenticity or the independence of an oracle. Valid finite prefixes can be scored without natural episode termination. Horizons after the observed prefix are censored; do not invent absorbing states or call env.step after termination to fill them. Report the denominator at every horizon and, when comparing horizon curves, also consider the same source cohort. Sources surviving longer can differ from those that terminate. forecast_metrics reports signed/absolute error in each coordinate’s physical unit. When standard deviations are supplied, it also reports nominal interval bounds, observed coverage and marginal Gaussian negative log density. Density is not probability; its logarithm depends on units and can be negative. Missing uncertainty stays unavailable. This API does not prove Gaussian assumptions or calibration, and it does not combine unlike physical units into a quality score.

Connect to release evidence

forecast_case_result(forecast, tolerances={...}) applies explicit per-coordinate absolute-error tolerances and produces normal saved trials. Checks are named dynamics/<coordinate>/h<horizon>. Censored checks are skipped. Require their coverage through an AcceptancePolicy; an aggregate pass rate alone can omit missing horizons. Simulator failures become evaluator errors with their source identified, while model failures remain model errors. Choose tolerances from the application’s failure consequences, not a universal world-model threshold. The bridge’s model ID is caller supplied: record the checkpoint/configuration digest and preserve upstream model artifacts yourself. The synchronous callback cannot interrupt a blocked prediction; use your existing execution framework for process isolation, deadlines and cancellation.

Run the learned-model demonstration

pip install -e '.[gymnasium,review]'
python benchmarks/industrial/world_model_experiment.py --smoke --output-dir /tmp/world-smoke
python benchmarks/industrial/world_model_experiment.py --output-dir /tmp/world-heldout
python benchmarks/industrial/analyze_world_model.py --replay /tmp/world-heldout

The frozen protocol reuses native Gymnasium dynamics, scikit-learn Bayesian regression and SciPy search. It includes action removal and state-persistence controls, matched interventions, uncertainty diagnostics and actual closed-loop execution. The results and raw evidence show why excellent next-angle prediction and state-offset persistence are insufficient evidence of planning usefulness. All measurements concern this small fully observed simulator experiment; no general or industrial capability claim follows from it.

Getting Started

Evaluators

Claude Code Skills

Guides

Reference

Compliance

Experimental world-model evaluation

Measure the capability you need

Capture an actual reference and forecast

Preserve missing evidence

Connect to release evidence

Run the learned-model demonstration

​Measure the capability you need

​Capture an actual reference and forecast

​Preserve missing evidence

​Connect to release evidence

​Run the learned-model demonstration

Measure the capability you need

Capture an actual reference and forecast

Preserve missing evidence

Connect to release evidence

Run the learned-model demonstration