suite.run(model_fn) calls model_fn(case.input)
for each case, so a model_fn that looks up the logged answer keyed on the input
is the supported path. Replay your log; score it.
The pattern
Build a map from each case’sinput to the answer you already logged, then hand
suite.run a lambda that reads from it:
model_fn; it returns the recorded output instead
of calling a live model. Every evaluator runs exactly as it would on a fresh
generation.
End to end
Notes
- The case
inputis the key. Make sure eachEvalCase.inputmatches the prompt you logged the answer under, exactly. AKeyErrorfrom the lambda means a case has no replay entry. - Carry the
context. Grounded evaluators likeFaithfulnessandHallucinationscore the logged answer against the case’scontext, so include the context each answer was produced against. - No live calls. Because the lambda never touches a model, replay scoring runs at evaluator speed and costs only the LLM-judge calls (deterministic evaluators are free).

