> ## Documentation Index
> Fetch the complete documentation index at: https://docs.multivon.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Experimental world-model evaluation

> Bind vector-state forecasts to actual environment transitions and test decision usefulness.

This experimental interface is available in PyPI 0.19.0.
The initial profile evaluates **fully observed numeric state vectors**. Video,
latent-state decoding, hidden-object memory and real-robot control are outside its
tested scope. Agent reasoning traces use different evaluators.

## Measure the capability you need

| Question                                  | Evidence to collect                                                                   |
| ----------------------------------------- | ------------------------------------------------------------------------------------- |
| Does the model respond to actions?        | Matched initial states, different actions, predicted versus observed effects          |
| Does prediction deteriorate with horizon? | Per-coordinate errors at specified steps, with source counts and missing endpoints    |
| Does state persist appropriately?         | Controlled initial-state changes and their observed later effects                     |
| Is uncertainty useful?                    | Empirical interval coverage **and width**, with the declared distribution assumptions |
| Does the model help decisions?            | The same planner/budget operating in fresh real environment instances                 |

These are established evaluation concerns, not new Multivon metrics. See the
[decision-focused position paper](https://arxiv.org/abs/2606.15032). Use upstream
video tools such as [WorldFoundry](https://github.com/OpenEnvision/WorldFoundry)
or [WorldModelBench](https://arxiv.org/abs/2502.20694) for their supported outputs;
their results do not establish this profile's control performance.

## Capture an actual reference and forecast

Install the existing Gymnasium extra:

```bash theme={null}
pip install -e '.[gymnasium]'
```

```python theme={null}
import gymnasium as gym

from multivon_eval import EvalCase
from multivon_eval.dynamics import capture_forecast
from multivon_eval.dynamics_metrics import forecast_metrics
from multivon_eval.integrations.gymnasium import capture_episode

actions = [0, 1, 0, 1]  # fix the complete plan before executing the reference
holder = {}

def factory():
    holder["env"] = gym.make("CartPole-v1")
    return holder["env"]

def interact(env, observation, info):
    for action in actions:
        _, _, terminated, truncated, _ = env.step(action)
        if terminated or truncated:
            break

reference = capture_episode(
    factory, interact,
    case=EvalCase("Predict these CartPole actions", case_id="cartpole:7", source_id="seed:7"),
    environment_id="CartPole-v1/native", observer_id="native-state/v1",
    observe=lambda: {"state": list(holder["env"].unwrapped.state)},
    seed=7, max_steps=len(actions),
)

def persistence(request):
    return {"states": [list(request["initial_state"]) for _ in request["actions"]]}

forecast = capture_forecast(
    reference, persistence, model_id="persistence/v1", contract="cartpole-vector/v1",
    coordinates={"x": "m", "velocity": "m/s", "angle": "rad", "angular_velocity": "rad/s"},
    horizons=[1, 4], planned_actions=actions,
)
print(forecast_metrics(forecast))
```

The model receives a detached object containing only `initial_state` and
`actions`. It returns exactly one vector per supplied action. It may also return
aligned `standard_deviation` vectors for marginal Gaussian diagnostics and a
portable `metadata` object describing the method. Positive finite deviations,
finite coordinates and matching dimensions are required. Do not attach future
truth through the callback's closure or global state.

`planned_actions` must match the actually executed prefix. Freeze that sequence
before simulation: its length must not disclose when the reference terminates.
If omitted, the API explicitly records `action_scope="observed_prefix"`; the
model then knows the reference-dependent sequence length. This mode is useful
for saved-prefix diagnostics but cannot claim a predeclared forecast experiment.

## Preserve missing evidence

`ForecastEvidence` retains the bound native episode, request, returned forecast,
model/contract IDs and errors. References with simulator/setup/cleanup failures
receive `simulator_error`, and the model is not called. Failed or malformed model
responses receive `model_error`. Neither becomes a quality score. A digest checks
content consistency, not authenticity or the independence of an oracle.

Valid finite prefixes can be scored without natural episode termination.
Horizons after the observed prefix are `censored`; do not invent absorbing
states or call `env.step` after termination to fill them. Report the denominator
at every horizon and, when comparing horizon curves, also consider the same
source cohort. Sources surviving longer can differ from those that terminate.

`forecast_metrics` reports signed/absolute error in each coordinate's physical
unit. When standard deviations are supplied, it also reports nominal interval
bounds, observed coverage and marginal Gaussian negative log density. Density is
not probability; its logarithm depends on units and can be negative. Missing
uncertainty stays unavailable. This API does not prove Gaussian assumptions or
calibration, and it does not combine unlike physical units into a quality score.

## Connect to release evidence

`forecast_case_result(forecast, tolerances={...})` applies explicit per-coordinate
absolute-error tolerances and produces normal saved trials. Checks are named
`dynamics/<coordinate>/h<horizon>`. Censored checks are skipped. Require their
coverage through an `AcceptancePolicy`; an aggregate pass rate alone can omit
missing horizons. Simulator failures become evaluator errors with their source
identified, while model failures remain model errors.

Choose tolerances from the application's failure consequences, not a universal
world-model threshold. The bridge's model ID is caller supplied: record the
checkpoint/configuration digest and preserve upstream model artifacts yourself.
The synchronous callback cannot interrupt a blocked prediction; use your existing
execution framework for process isolation, deadlines and cancellation.

## Run the learned-model demonstration

```bash theme={null}
pip install -e '.[gymnasium,review]'
python benchmarks/industrial/world_model_experiment.py --smoke --output-dir /tmp/world-smoke
python benchmarks/industrial/world_model_experiment.py --output-dir /tmp/world-heldout
python benchmarks/industrial/analyze_world_model.py --replay /tmp/world-heldout
```

The [frozen protocol](https://github.com/multivon-ai/multivon-eval/blob/main/benchmarks/industrial/WORLD_MODEL_PROTOCOL.md)
reuses native Gymnasium dynamics, scikit-learn Bayesian regression and SciPy
search. It includes action removal and state-persistence controls, matched
interventions, uncertainty diagnostics and actual closed-loop execution.
The [results and raw evidence](https://github.com/multivon-ai/multivon-eval/blob/main/benchmarks/industrial/WORLD_MODEL_RESULTS.md)
show why excellent next-angle prediction and state-offset persistence are
insufficient evidence of planning usefulness. All measurements concern this
small fully observed simulator experiment; no general or industrial capability
claim follows from it.