Section 01
Prediction → Reasoning → Simulation
Percy Liang's framing of AI's progression is useful for founders: we had the age of prediction (classify this image), then the age of reasoning (solve this multi-step problem), and now the age of simulation — answering questions whose answers emerge from many humans interacting over time.
His examples are institutional: how would remote work change our culture? How should a national curriculum change? But the startup versions are the same question at smaller scale, and they're the questions founders ask every day:
- "If we price at $49 instead of $29, what happens to who buys?"
- "If we reposition from 'project management' to 'time tracking for agencies', who adopts and who churns?"
- "If we launch this feature for enterprise first, how does the SMB segment react when it reaches them?"
These are intervention questions — in Pearl's causal hierarchy, they sit above prediction. A chatbot can speculate about them. A simulation can play them forward with a population model and show you the distribution of outcomes.
Section 02
What Simulation Unlocks That Chatbots Don't
The distinction Simile draws — and we've built our product around — is that simulation produces an auditable artifact: a population model, a scenario, and a distribution of outcomes you can inspect, question, and stress-test. A chatbot produces a plausible paragraph.
| Capability | ChatGPT-style LLM | Population simulation |
|---|---|---|
| 'Would SMBs buy this?' | One plausible opinion | Distribution across 1M+ persona segments |
| Consistency across questions | Drifts with each prompt | Persona attributes locked (income, role, location) |
| Segment comparison | Manual prompting per segment | Parallel panels, side-by-side distributions |
| Confidence in the answer | None | Stability scores + benchmark-calibrated error estimates |
| Cost per question | Pennies (but unreliable) | Minutes + dollars, with error bars |
Input
- Question
- "Would you pay $49/mo for this?"
- ChatGPT answer
- One plausible paragraph, new every time
- Simulation
- Distribution across segments
- Confidence
- Stability TVD + red flags per question
Output
The segment split is the insight an average destroys. Simulation shows you not just '41% yes' but which 41% — and whether the segments disagree.
Section 03
The Startup Playbook for Simulation
Enterprises use simulation to de-risk nine-figure launches. Startups can run the same loop at their scale. The playbook we see working:
- Simulate the intervention, not the product. Don't ask 'do you like my app?' Ask 'your agency just switched to this — what changes in your week?' Interventions produce behavioral distributions; descriptions produce polite nods.
- Read distributions, not averages. A 3.2 average can hide a segment at 4.5 and a segment at 1.8. The network visualizer exists because the shape of disagreement is the insight.
- Check the confidence badge before the conclusion. A stable panel (predicted TVD ≤ 0.10) supports directional decisions. A flagged panel means re-run with better targeting, not 'the market said no'.
- Simulate → validate → calibrate. Use simulation to narrow 10 options to 2, spend real research on the 2, and compare what the panel predicted against what humans said. That calibration loop is how simulation gets more trustworthy for YOUR market over time.
Section 04
How We're Tackling It: Confidence, Stability, and Adherence
Everything above assumes you can tell a trustworthy panel from a broken one. That's the hard part — so we built it into the product rather than leaving it to faith. Every TestSynthia survey now ships with two automatic verification layers that run at the end of the analytics pipeline:
| Layer | Question it answers | How it works |
|---|---|---|
| Stability score | Does the panel agree with itself? | Split-half TVD: the panel is randomly split in two, 20 times, and we measure Total Variation Distance between the halves' opinion distributions |
| Persona adherence score | Is each persona actually acting like their profile? | A strict LLM auditor checks each persona's answers against declared traits — a 30-year-old dev should talk like a dev, a 65-year-old shouldn't call themselves a digital native |
| Benchmark TVD | Does the aggregate match real humans? | Ongoing calibration against published survey distributions (GSS 2024), with noise-floor math separating sampling error from model bias |
The stability score is deterministic math, not vibes: zero LLM calls, computed in milliseconds, seeded so retries produce identical values. The result maps to a bucket shown on every survey:
Stable panels have hit split-half TVD as low as 0.05 — random halves agree within 5 points. The 0.16 decision-grade bar comes from Simile's published calibration.
- Stability is a lower bound, and we say so. A panel from one LLM can be consistently wrong together — high stability is necessary for accuracy but not sufficient. That's why the benchmark program exists: to measure the stability→accuracy relationship against real human data.
- Adherence closes the 'stable but fake' gap. A perfectly stable panel can consist of personas all acting like someone they're not — every cross-tab and segment analysis would be quietly wrong. The adherence layer catches this per-persona, and thin or empty responses are counted as failures so high adherence can't mask a broken generation run.
- The confidence badge makes the decision rule explicit. Green: proceed with directional decisions. Yellow: review the flagged questions listed on the badge. Red: don't decide on this panel — re-run with better targeting. The failure mode 'the market said no' from a broken panel is exactly what this is designed to prevent.
Section 05
Why This Moment Is Different
Simulated personas existed as research demos for years. Three things changed in 2025–2026 that make them decision infrastructure:
- Inference cost collapse. Wafer-scale (Cerebras) and optimized serving (BaseTen) made 1M-persona runs economically viable. We documented our own 70% generation-time cut in an earlier post.
- Confidence research. Simile's published work on predicting simulation error turned 'trust me' into measurable error bars — and gave the whole field a target metric (TVD < 0.16).
- Enterprise validation. CVS Health running generative-agent simulations for patient-experience decisions means the methodology has crossed from experiment to practice.
The bottleneck has moved upstream — from 'can I build it?' to 'should I, for whom, and what happens next?' Those are simulation questions. The startups that learn to ask them will iterate at a cadence competitors relying on gut feel can't match.
FAQ