PIIEvaluator
Scans LLM outputs for personally identifiable information using local regex patterns plus checksum validation (Luhn, Verhoeff, Mod-97, Mod-11). Zero API calls by default. Optional NER fallback via Presidio when installed. When to use: Any regulated deployment (healthcare, finance, legal, government) where PII in model outputs is a compliance risk. Run it in CI/CD so regressions are caught before production. Passes when no PII is detected. Fails with a per-type breakdown.Parameters
Standards covered
Every pattern in the evaluator carries a citation to its source standard. The table below is exhaustive — if your jurisdiction needs an identifier not listed here, supply it viapatterns={...} (or open a PR — additions
are easy).
HIPAA Safe Harbor — 45 CFR § 164.514(b)(2)
Eighteen identifier categories. The evaluator covers thirteen via regex; the remaining five (free-text names, geographic subdivisions, photographs, biometrics, “other unique IDs”) needuse_ner=True for partial coverage.
GDPR — Regulation (EU) 2016/679, Art.4(1)
National identification numbers across EU member states + base PII.Article 9 special categories (race, religion, health data, sex life, trade-union membership): these are content categories, not identifier formats. Use a topic classifier or NER pipeline; the regex evaluator can’t reach them.
DPDP India — Act 22 of 2023
Indian government-issued identifiers + Indian PII formats.CCPA — Cal. Civ. Code § 1798.140(o)
(Other CCPA categories — biometric data, geolocation, browsing history —
need pipeline-level controls; regex can’t cover them.)
PIPEDA (Canada)
Schedule 1 categories overlap entirely with the base PII set (name, email, phone, address, SSN/SIN, financial). No Canada-specific identifier format needs to be added.Strict mode (default)
Whenstrict=True (the default), regex hits are filtered through identity
validators before being reported. This dramatically cuts false positives:
Pass
strict=False to see raw matches without validation — useful for
debugging and for jurisdictions where checksum specs aren’t published.
Optional NER (use_ner=True)
When use_ner=True, the evaluator additionally invokes
presidio_analyzer on the output to
catch PERSON, LOCATION, DATE_TIME, NRP, ORGANIZATION, etc. — providing
partial coverage for HIPAA Safe Harbor categories that regex can’t reach
(unprefixed names, free-form addresses, biometrics references).
Presidio is an optional dependency:
use_ner=True is a silent no-op — the
evaluator just runs the regex/checksum pipeline.
Sample output
strict=True, decoys like 1234-5678-9012 (not Verhoeff-valid
Aadhaar), 4532-0151-1283-0367 (Luhn-invalid Visa shape), and
123-45-6789 (test SSN) are dropped from the report.
SchemaEvaluator
Validates that LLM outputs conform to a Pydantic model or JSON Schema dict. Zero API calls — validation is purely local. When to use: Structured output tasks — extraction, classification, API response generation — where you need per-field failure breakdowns rather than a binary pass/fail. Passes when output is valid JSON matching the schema. Fails with per-field error messages.strict mode behavior: When strict=False (default), extra keys in the JSON output are ignored — only required fields and type constraints are checked. When strict=True, any key present in the output that is not declared in the schema is counted as a violation. Use strict mode when you need to enforce that the model doesn’t leak internal fields or hallucinate extra properties.
Strips markdown code fences automatically before parsing. For JSON Schema, scoring is proportional: score = max(0.0, 1.0 - errors/10). For Pydantic models, any validation error returns score 0.0.
Sample output: