Section 01
The Gap: Two Ways a Panel Can Be Wrong
Our stability score answers one question: does the panel agree with itself? Split-half TVD catches noisy panels — but a panel can be perfectly stable and still wrong, because every persona in it might be consistently behaving like someone they're not.
Consider a panel where every 'software developer' persona gives answers indistinguishable from a marketing manager's. The aggregate distribution might still look plausible. Split-half TVD stays low. Nothing flags it. But every response came from a persona that failed to be who the sampling said it was — which means the demographic cross-tabs, the segment analysis, and the JTBD breakdown are all quietly wrong.
This is the 'good personas, wrong population' trap from the LLM-S3 benchmark literature — and it has an inverse: 'consistent personas, fake population.' You need two separate measurements:
| Check | Question it answers | Catches |
|---|---|---|
| Stability (split-half TVD) | Does the panel agree with itself? | Sampling noise, incoherent personas |
| Adherence (this post) | Does each persona act like their profile? | Persona rendering bugs, prompt drift, broken persona blocks |
| Benchmark TVD | Does the aggregate match real humans? | Composition error, systematic bias |
Section 02
The Method: Declared Traits, Audited Answers
The design borrows from two research threads. MatrAIx — a system simulating 8.3 billion persona agents across 1,290 dimensions — validated their personas by checking whether declared behavioral traits were expressed in 18,189 evaluation trials (91.5% adherence). Potham's 2025 adherence benchmark contributed the scoring discipline: unambiguous violation conditions, positive-framed checks ('responses SHOULD reflect X' rather than 'must not contradict X'), and a critical warning: high adherence can mask incompetence — a persona giving empty, generic answers passes every trait check vacuously.
For each persona in a panel, we derive a set of declared behavioral traits directly from their stored profile — the same fields that feed the generation prompt:
| Persona attribute | Derived trait | What the auditor checks |
|---|---|---|
| Occupation: software developer | tech_savvy | Answers show tech fluency, not confusion about apps |
| Age: 19 | early_adopter_comfort | Comfort with modern digital products |
| City: San Francisco | urban_lifestyle | City-life references (rent, commute, transit) — not rural imagery |
| Education: high school | no_jargon_fluency | No confident consulting jargon (ROI, unit economics) |
| Sports persona: daily gym | active_lifestyle | Fitness surfaces naturally in their reasoning |
A strict LLM auditor (temperature 0, JSON verdicts) then reads each persona's actual survey responses and issues a three-state verdict per trait: expressed (evidence supports it), no-opportunity (the survey never touched a topic where the trait could show), or contradicted (the responses actively conflict — a self-described home cook saying 'I never cook, hate kitchens'). Only a contradiction fails a persona.
Section 03
Tiered Scoring: Not All Traits Are Equal
A persona contradicting its age is a worse failure than a persona not mentioning its gym habit. So traits carry weights, and the per-persona score is weighted:
| Tier | Weight | Traits | Detection |
|---|---|---|---|
| Tier 1 — Contradiction | 3.0 | Age identity, gender consistency, occupation jargon ceiling, education ceiling | Deterministic regex extraction (claimed ages, spouse references) + LLM judge on explicit factual claims only |
| Tier 2 — Voice | 2.0 | Tech fluency, hands-on work grounding, education-appropriate language, early-adopter comfort | LLM judge |
| Tier 3 — Identity | 1.0 | Urban lifestyle, cooking, travel, fitness, location anchor | LLM judge — only when the survey context invites the topic |
Tier 1 is partly deterministic: regex patterns extract explicit claims from free text ('I am 28', 'my wife…') and match them against declared demographics, plus a jargon-blocklist check (a fast-food worker confidently discussing SIEM architecture is a hard contradiction). Deterministic checks can't be fooled by judge leniency, and they're free.
Section 04
What It Caught — and What It Shouldn't
We validated the layer on two real panels: a 60-persona Pew AI benchmark panel and a 48-persona B2B security panel. After fixing the expressibility bug and tightening trait definitions, both landed at 93.3% and 91.7% — right around MatrAIx's published 91.5%.
Input
- Panels validated
- 60 + 48 personas (two surveys)
- Traits checked
- ~4 per persona (tiered, weighted)
- Judge
- Strict LLM auditor, temperature 0
- Cost control
- Panels > 100 judged on a seeded random sample
Output
Adherence rates after fixing the expressibility bug and tuning trait definitions. One genuine outlier persona (failing 5 of 6 traits) was caught and is exactly the failure mode this layer exists to surface.
The remaining failures were inspectable and genuine: personas whose answers contradicted their declared cooking habits, high-school-educated personas confidently using consulting jargon, and one multi-trait outlier. That's what a trustworthy QA metric looks like — the failures themselves tell you where to look.
Section 05
Where This Fits in the Verification Stack
Every TestSynthia survey now runs a verification stage at the end of the analytics pipeline. Both checks execute in parallel and both surface as compact color-coded badges on the survey page:
- Stability badge (shield icon): the split-half TVD agreement % — green/yellow/red by bucket, with a special blue 'small panel' state below 50 personas where the math is directional.
- Adherence badge (persona icon): the share of personas passing trait validation, with the same color coding. Hover either badge for the full mechanics.
The roadmap from here mirrors the benchmark program: adherence failures by trait are logged per survey, so we can measure whether persona-rendering changes move the needle, and eventually feed trait-level failure rates into the calibrated confidence model — the same trajectory Simile documented from features to fine-tuning.
“A panel that agrees with itself is not enough. A panel that matches published distributions is not enough. The personas themselves have to be real people before either of those numbers means what you think it means.”
FAQ