Prompt attribution

​

When to use it

• In CI, on every PR: run attribution diff base head and post the Markdown output as a PR comment. Reviewers see exactly which prompts changed without scrolling the unified diff.
• From /eval-audit: the skill calls scan to identify which call sites the PR touched, then runs only the eval cases that stress those surfaces.
• One-off audits: attribution scan . lists every prompt literal in your codebase. Useful when onboarding to a repo you didn’t write.

​

What it captures

# anthropic
anthropic.messages.create(system="<literal>", messages=[{"role": "user", "content": "<literal>"}])
client.messages.create(...)              # any *.messages.create

# openai
openai.chat.completions.create(messages=[...])
client.chat.completions.create(...)      # any *.chat.completions.create

# litellm
litellm.completion(messages=[...])
litellm.acompletion(messages=[...])

• Plain string literals: system="..."
• f-strings with zero runtime interpolation (their content is fully known at parse time)
• Module-level string constants referenced by name at the call site (one hop, same file; see below)

​

Scanner v3 (0.10.1)

• Aliased litellm imports (import litellm as llm, from litellm import completion as c) now resolve to litellm call sites.
• **kwargs-unpacked calls are recorded as dynamic records with placeholder <dynamic:KwargsUnpack> instead of being invisible.
• messages=<variable> (the dominant real-world pattern) produces an honest dynamic record instead of nothing. A statically-known empty messages=[] still produces nothing — there is no prompt there.
• One-hop module-constant resolution. SYSTEM_PROMPT = "..." at module scope followed by system=SYSTEM_PROMPT at a call site now resolves to the real text instead of a dynamic placeholder. The rules are deliberately conservative, because a false “static” poisons every downstream verdict while a false “dynamic” is merely cautious. A name assigned more than once, assigned under a conditional/loop/try, augmented, tuple-unpacked, shadowed in the enclosing scope, declared global, or defined in another module stays dynamic. One hop only: X = Y chains do not resolve.
• loose_fingerprint on every record: a hash of the whitespace-collapsed prompt text, alongside the strict fingerprint. It exists solely to label a change as formatting-only, never to suppress it.

​

Scanner v4 (0.11.1)

• Fingerprints are NFC-normalized (the version bump). Composed vs decomposed unicode (“é” as one codepoint vs e + combining accent) is an editor/OS artifact, not a prompt change — it previously fingerprinted as drift.
• match-statement capture patterns disqualify module constants. case PROMPT: rebinds the name through a path the constant resolver didn’t see, letting a rebound constant read as static. Such names now stay dynamic.
• Unscannable files surface as UNSCANNABLE, never REMOVED. A file the scanner can’t parse (syntax error, non-UTF8 encoding) previously returned zero records, so the staleness report marked every baselined site in it REMOVED, and --fail-on removed failed CI with a misleading verdict. They now report as a distinct UNSCANNABLE tier (“file exists but could not be parsed — verdict unknown, NOT removed”): a warning names each file with its reason in all three renderers, the JSON report gains skipped_files, and --fail-on removed no longer trips. Skipped files are a report-time concept, never written into baselines.
• Symlinks resolving outside the repo root are skipped, not recorded. They previously wrote machine-specific absolute paths into the baseline, producing false REMOVED+ADDED churn on every other checkout.

​

attribution scan

multivon-eval attribution scan ./my-repo

multivon-eval attribution scan ./my-repo --format json > prompts.json

Found 7 prompt(s) across 3 file(s):

  src/agent/planner.py:42:anthropic.system
      qualname=Planner.build_prompt  fp=a3f1b2c8d4e0…
      first line: 'You are a careful planning agent.'
  src/agent/planner.py:78:anthropic.user#0
      qualname=Planner.build_prompt  fp=9c2e7a1f6d3b…
      first line: 'Plan the following task step by step:'
  src/extractors/invoice.py:91:anthropic.system  [dynamic]
      qualname=extract_invoice  fp=000000000000…
  ...

​

attribution diff

multivon-eval attribution diff ./repo-base ./repo-head

multivon-eval attribution diff ./repo-base ./repo-head --format text

multivon-eval attribution diff ./repo-base ./repo-head --format json

​

How identity works

​

change_type values

​

Programmatic API

from multivon_eval.attribution import (
    scan,
    diff_records,
    render_markdown,
)

base = scan("/path/to/repo-base")
head = scan("/path/to/repo-head")
diffs = diff_records(base, head)
md = render_markdown(diffs)
print(md)

​

Wiring it into CI

- name: Prompt diff
  run: |
    multivon-eval attribution diff ./base ./head --format markdown \
      > prompt-diff.md
- uses: marocchino/sticky-pull-request-comment@v2
  with:
    path: prompt-diff.md

​

What’s next

• The Phase 2 sidecar that gates causal attribution on a non-prompt-change signal is under design — see the package docstring in attribution/__init__.py.
• Tier-2 identity (file-move / line-shift detection) is on the roadmap; until it lands, a refactor that moves a prompt to a different line will show up as removed + added.

​

See also

• Prompt-drift staleness — the standing drift report built on this scanner: baseline snapshot, case provenance, CI gate.
• /eval-audit skill — the PR-gating workflow that consumes attribution scan to scope which cases to re-run.
• eval-action on GitHub — the GitHub Action that wraps this CLI plus the suite runner.
• Bootstrap workflow — generate the suite that /eval-audit runs.
• Source on GitHub — five small files, ~400 lines total.