Synthetic research needs a higher standard

Market research has always involved a trade-off.

Good human research is invaluable, but it can be slow, expensive and hard to run at the speed modern teams make decisions. By the time a survey is recruited, fielded, analysed and turned into recommendations, the product, message or strategy may already have moved on.

So teams make compromises. They test fewer ideas. They rely on instinct. They put concepts in front of customers only after expensive work has already begun. They ask researchers to do more with less.

Large language models create a new possibility: a way to ask more questions, earlier, and learn faster from research evidence that already exists.

But there is a problem. It is easy to ask an AI model to “act like a 35-year-old parent in Melbourne” and generate a plausible research response. Plausible is not the same as valid. A synthetic respondent can sound convincing while being too polished, too average, too certain, too socially desirable, or too dependent on demographic stereotypes.

That is why we believe the first generation of “AI personas” is not enough.

The next generation of synthetic research needs to be grounded in real human data, transparent about uncertainty and validated against answers real people actually gave.

That is the category we are building.

Synthetic research should not be AI roleplay. It should be a rigorous inference layer built on observed human evidence.

First principles: what large language models actually are

A large language model, or LLM, is a machine learning system trained on extremely large collections of text and other data. During training, it learns statistical patterns in language: which words, concepts, arguments, emotions, explanations and decisions tend to appear together in context.

The simple explanation is that an LLM learns to predict what should come next in a sequence.

The important implication is larger: to do that prediction well, modern models learn rich internal representations of language, categories, reasoning patterns, social situations, preferences, trade-offs and intent. They can read a product concept, understand the differences between two answer options, infer what kind of evidence is relevant, summarise messy open-ended responses, translate between natural language and structured data, and generate human-like explanations.

That is why LLMs feel powerful in research contexts. Research is full of language and judgement:

• survey questions and answer options
• brand associations
• product concepts
• open-ended responses
• customer complaints
• reasons for choice
• emotional reactions
• barriers and motivations
• contradictory attitudes
• segment descriptions
• strategic hypotheses

LLMs are unusually good at working across all of that material.

But an LLM is not a magic respondent. It does not automatically know what your customers, voters, members or audience think. It does not contain a live panel of people. It can hallucinate. It can overgeneralise. It can give a confident answer when the available evidence is weak.

So the question is not whether LLMs can produce realistic-sounding research responses. They can.

The better question is:

Can we constrain these models with real respondent evidence, measure uncertainty, and validate the outputs against human ground truth?

That is the methodological standard we are building around.

Why LLMs are especially powerful for market research

Market research has always tried to answer a difficult question:

What would a real person think, feel or do in response to something they have not yet seen?

That could be a new product, a policy message, a pricing change, a brand idea, a campaign line or a customer experience.

Traditional research answers that by asking new people. Synthetic research asks whether we can also infer useful directional answers from people whose relevant attitudes, behaviours and preferences have already been measured.

LLMs matter because they make that inference more flexible.

Older statistical models are strong when the target question closely resembles historical variables and the data is clean and structured. They are weaker when the new stimulus is natural language: a paragraph-long concept, a campaign message, a product description, a claim, a positioning statement, or a messy qualitative prompt.

LLMs can understand the new stimulus itself. They can compare it with observed respondent evidence. They can reason across structured survey fields and unstructured text. They can express why a response may be positive, negative, mixed or uncertain.

Used carefully, this unlocks a new research workflow:

• test more concepts before committing budget
• explore more segments before fielding a full study
• identify hypotheses worth validating with humans
• pressure-test messages against known attitudes
• use existing survey assets more deeply
• turn respondent-level datasets into reusable insight infrastructure
• move from one-off reports to living research models

This is the commercial promise of synthetic research.

The catch is that this promise only holds if the system is built like research software, not like a persona-writing toy.

The problem with generic AI personas

Most people are right to be skeptical of synthetic research.

If a platform creates a fictional person from a few demographic fields, then asks an LLM to imagine what that person would think, the result may be useful for brainstorming. It should not be mistaken for research-grade evidence.

The risk is not that the output looks bad. The risk is that it looks too good.

Generic AI personas can produce fluent explanations, neat motivations and coherent attitudes. Real people are not always like that. They contradict themselves. They care about sustainability but buy the cheaper option. They say a brand matters, then switch for convenience. They are unsure, distracted, inconsistent and influenced by context.

A methodology that smooths all of that into polished copy can mislead decision-makers.

A more rigorous approach has to answer the hard questions skeptics ask:

• What real evidence grounded this answer?
• Was the response inferred from an individual human record or from a stereotype?
• How uncertain was the prediction?
• Does the synthetic population preserve the distribution of the real one?
• Has the method been backtested against hidden human answers?
• Does the system know when the source data is not strong enough to support a question?

Those are the questions our methodology is designed around.

Our position: not an AI panel, an empirical research engine

We do not think the strongest version of synthetic research is “ChatGPT pretending to be consumers”.

We think it is a new research layer that sits on top of real respondent-level data.

Our platform starts with human microdata: survey responses, respondent attributes, codebooks, labels, weights and, where available, longitudinal or behavioural history. Each synthetic respondent model is anchored to an actual respondent record from that source data.

If a dataset contains 2,070 respondents, we start with 2,070 respondent models. We do not need to invent 10,000 fictional people to make the work sound more impressive.

The language model has an important role, but it is not the empirical foundation.

The empirical foundation is the human data.

The model's job is to infer how a specific respondent is likely to answer a new question, conditional on what that respondent has already told researchers.

That distinction changes the product from a persona generator into a research inference system. It also changes what clients should expect: not magic, not certainty, not a replacement for every human study, but a faster way to explore questions using the evidence already embedded in real research data.

The category opportunity: synthetic research as an insight acceleration layer

We believe synthetic research will become a standard part of the modern research stack.

Not because it replaces human respondents in every context. It will not.

It will matter because teams have far more questions than they can afford to field. Every brand, product, policy and growth team has a backlog of ideas they would test if research were faster and more flexible.

Synthetic research is best understood as an insight acceleration layer:

• before human fieldwork, to prioritise what is worth testing
• between waves, to explore follow-up questions the original survey did not ask
• after a study, to get more value from respondent-level data
• during strategy development, to compare concepts quickly
• during message development, to identify likely segment reactions
• during early product discovery, to expose risks and hypotheses

This does not make human research less important. It makes human research more focused.

Instead of using expensive fieldwork to test every rough idea, teams can use synthetic inference to narrow the field, sharpen the stimulus, identify uncertainty and decide where human validation will matter most.

In that sense, the future is not synthetic versus human research.

It is synthetic plus human research, with each used where it is strongest.

How our methodology works

At a high level, our methodology follows this flow:

1. Real human research data
2. One respondent model per real respondent
3. Raw observed response history preserved
4. New question analysed
5. Question-relevant human evidence retrieved
6. Independent probabilistic inference
7. Repeated runs measure instability
8. Source survey weights applied
9. Population estimate and uncertainty
10. Held-out human answers used for validation

Each step is designed to make synthetic research more defensible.

1. We anchor every respondent model to real human data

Our core unit is not a fictional persona. It is a respondent model anchored 1:1 to a real respondent in an underlying research dataset.

For each respondent, we preserve the observed source evidence:

• demographic and contextual fields, where available
• the questions they were asked
• the answers they gave
• response labels and codebook metadata
• missingness, refusals and “don't know” responses
• survey weights, where provided

We deliberately avoid making a polished biography the source of truth.

A summary like “Sarah is a progressive, environmentally conscious mother” may be easy to read, but it loses the actual structure of the evidence. Did Sarah vote progressive once, identify strongly with that party, prioritise climate policy, support environmental regulation, or merely live in a suburb where that is common? Those differences matter.

The source of truth remains the respondent's observed research history.

The LLM is then used to interpret that evidence in relation to a new question.

2. We use LLMs where they are strongest: interpreting context

The power of an LLM in synthetic research is not that it can invent a believable person.

The power is that it can interpret a new research stimulus in light of a specific respondent's prior evidence.

For example, a traditional survey dataset may not have asked:

Would this household consider switching to a subscription grocery service if it saved 15 minutes per week but cost $12 more?

But the dataset may contain relevant evidence:

• household composition
• income band
• shopping frequency
• convenience orientation
• price sensitivity
• past subscription behaviour
• attitudes to time pressure
• trust in online services
• open-ended frustrations with grocery shopping

An LLM can understand the new concept, retrieve and weigh the relevance of those observed fields, and estimate a response distribution for that particular respondent.

This is where LLMs are genuinely useful: not as imaginary consumers, but as flexible reasoning engines constrained by empirical data.

3. We retrieve question-relevant evidence instead of stuffing every field into a prompt

Real research datasets can contain hundreds or thousands of variables. More context is not automatically better context.

If the new question is about electric vehicle consideration, the respondent's views on transport, environmental issues, technology adoption, household circumstances, car ownership and price sensitivity may be highly relevant. An unrelated answer about television viewing may not be.

For each new question, our system creates a question-specific grounding bundle:

stable respondent context + relevant observed response history + the new question

The underlying raw research record remains intact. Retrieval is an evidence-selection layer, not a replacement for the respondent.

We record the exact source fields supplied to each inference. This creates an auditable evidence trail and lets us test a basic empirical question: how much and what kind of respondent history improves prediction?

4. We estimate probabilities, not just forced answers

Humans are uncertain. Synthetic respondents should be allowed to be uncertain too.

Imagine a respondent whose history suggests they lean towards a brand but have mixed views about price, trust and convenience.

A one-shot system must convert that ambiguity into a hard answer:

Yes

Our quantitative engine instead estimates a response distribution:

Yes: 0.68 · No: 0.32

For a five-point purchase-intent scale, the model estimates a probability across all five valid options.

The complete distribution is retained. If a study needs a single rendered response for an individual respondent, that answer can be sampled from the distribution. But the underlying uncertainty is not discarded.

This is important at population level. If thousands of ambiguous respondent models are each forced into their single most likely answer, a synthetic sample can become artificially certain. Preserving probabilities lets individual uncertainty contribute to the aggregate estimate.

5. We repeat inference to measure instability

A single model response can be fragile.

LLM outputs can be sensitive to prompt wording, answer-option structure and model configuration. A serious synthetic research platform needs to measure its own instability rather than hide it.

For standard quantitative studies, we can run multiple independent inferences for the same respondent and question using the same empirical grounding. The runs do not see one another's answers.

We then aggregate those runs into a respondent-level response distribution and calculate measures such as:

• top-answer agreement
• response entropy
• run-to-run dispersion
• respondent stability

A respondent whose independent runs repeatedly favour the same answer is different from one whose result changes frequently.

At study level, we can report the average stability of respondent models and the share of the population with high inference uncertainty.

We call this simulation uncertainty. We do not present it as a conventional survey margin of error, because it measures a different source of uncertainty.

6. We preserve the source population and apply survey weights after inference

A common synthetic-research pattern is to generate or resample a large synthetic population before asking questions.

Where we already have a respondent-level source dataset, we take a different approach.

We infer a response for every eligible respondent model in the source bank. We then aggregate the response probabilities using the survey weights supplied by the source study.

For response option k:

P(k) = ( Σ wᵢ · pᵢ(k) ) ⁄ ( Σ wᵢ )

where wᵢ is respondent i's source survey weight and pᵢ(k) is that respondent model's estimated probability of choosing option k.

This preserves the source survey's population structure without pretending that generating 10,000 synthetic draws creates 10,000 new independent human observations.

Our reporting distinguishes between:

• the number of source respondent models evaluated
• the number of model inference runs executed
• any optional simulated responses rendered from the resulting distributions

When survey weights are used, we also retain the weighting methodology and can report the effective weighted sample size.

7. Quantitative inference and qualitative expression are separate jobs

Large language models are exceptionally good at language. That is also one of the risks in synthetic research.

A generated respondent may give a polished, psychologically coherent explanation that the underlying human never would have articulated. The explanation can sound persuasive even when the quantitative judgement is unstable.

We therefore separate two functions.

Quantitative inference

For binary, single-choice, Likert and other structured questions, the system estimates a probability distribution over the permitted answers.

The output is constrained and machine-readable. It includes references to the observed respondent fields used as evidence.

We do not require the model to produce a long rationale or reveal private chain-of-thought.

Qualitative expression

For open-ended reactions and follow-up questions, a separate qualitative engine can express a synthetic response grounded in the same observed evidence.

Where a qualitative question follows a structured answer, the process is response-first and explanation-second. For example:

1. the respondent model's purchase-intent response is estimated
2. the qualitative engine is given that response and the supporting respondent evidence
3. it generates a plausible synthetic explanation consistent with both

This reduces the risk that a beautifully written explanation silently determines the quantitative result.

Generated qualitative responses are always labelled as synthetic responses. They are not represented as words spoken by the original human respondent.

8. Every answer has an evidence trail

Auditability matters more to us than theatrical “thinking”.

For each respondent and question, we persist the grounding bundle: the stable respondent context and the exact observed survey responses supplied to the model.

Our quantitative outputs can reference those source evidence fields.

That means a researcher can inspect why a result was well grounded without being shown an invented psychological monologue.

For example, a voting-intention inference may reference observed fields covering:

• previous vote
• party identification
• ideological self-placement
• issue priorities
• trust in government

The evidence trace says what observed human data grounded the inference. It does not claim to reveal the respondent's true internal reasoning.

9. We validate against answers real people actually gave

The most important part of our methodology is validation.

A synthetic respondent should not be called “realistic” because its answers sound realistic. It should be tested against human ground truth.

Our validation framework uses held-out response prediction.

For an existing research dataset, we can:

1. hide a real answer from a respondent model
2. remove that answer from all grounding and retrieval paths
3. build the respondent model from the remaining observed history
4. ask the held-out question
5. compare the synthetic prediction with the real answer

We repeat this across respondents and questions.

This produces objective measures of respondent-model performance. Depending on question type, these include accuracy, balanced accuracy, mean absolute error, rank correlation, Brier score and log loss.

We also test probability calibration. If the system assigns roughly 70% probability to an answer across many predictions, a well-calibrated engine should be correct at approximately that rate.

Crucially, we compare new engine versions with simple baselines and with our previous methodology. Complexity is not evidence of improvement.

The validation target is straightforward:

Does this architecture predict hidden human answers better, and more reliably, than simpler alternatives?

10. We measure population fidelity separately from respondent fidelity

Getting many individuals approximately right does not guarantee that the synthetic population is right.

A system could generate plausible individual answers while compressing the population towards moderate, agreeable or socially acceptable responses.

For that reason, our validation has two distinct levels.

Respondent fidelity

How well does the system predict held-out answers for individual real respondents?

Population fidelity

How well does the synthetic distribution preserve the patterns in the real human data?

Population-level tests can examine:

• marginal response distributions
• variance
• rare and tail responses
• subgroup differences
• pairwise associations and correlations
• joint distributions

This distinction is important because headline averages can look plausible even when the shape of the population is wrong.

A credible synthetic research methodology must test both the people and the population.

11. We assess whether a question is researchable from the available data

Synthetic inference is not equally suitable for every question.

Consider two questions asked of a political survey dataset.

Question A — “If an election were held today, would you vote first-preference Labor?”

The source data may contain previous vote, party identification, ideological self-placement, issue priorities and government trust.

Question B — “Would you prefer a new strawberry-flavoured biscuit to the original?”

The same dataset may contain no evidence about food, flavour or product preferences.

A language model can answer both. Our methodology should not treat those answers as equally supported.

Before a study runs, our platform assesses researchability: how strongly the available respondent evidence and the platform's historical validation performance support inference on the new question.

The score considers factors such as:

• relevance of available source questions
• depth and coverage of respondent evidence
• the share of respondent models with adequate grounding
• validation performance on similar question domains
• validation performance for the requested response type
• distance from the source dataset's observed domains

A high researchability score means the question is well supported by the available respondent evidence and similar inference tasks have validated well.

A low score triggers a warning:

Low synthetic researchability. Treat results as exploratory; human fieldwork is recommended.

This may be the most important feature in a responsible synthetic research platform: the ability to say we do not have enough evidence to infer this well.

A skeptic's concern, and our answer

Every hard question a skeptic asks has a direct methodological answer:

• “Isn't this just ChatGPT making things up?” — No. Each respondent model is anchored to a real respondent record, and inference is constrained by observed survey evidence.
• “Won't the AI just stereotype people by demographics?” — Demographics are only one part of the evidence. We prioritise actual observed attitudes, behaviours and prior responses.
• “How do I know what evidence was used?” — Every inference stores a grounding bundle showing the exact source fields supplied to the model.
• “What if the model is uncertain?” — We estimate probabilities, repeat inference and report stability rather than hiding uncertainty behind a forced answer.
• “What if the synthetic population drifts from the real population?” — We test population fidelity separately from individual fidelity and apply source survey weights after inference.
• “Can this replace all human research?” — No. It is strongest as an insight acceleration layer, hypothesis generator and way to extend existing research. Low-support questions should go to human fieldwork.
• “How is performance proven?” — We backtest by hiding real answers, asking the respondent models to predict them, and comparing outputs with human ground truth.

What this gives research teams

When built rigorously, synthetic research can change the rhythm of decision-making.

Instead of saving research only for the biggest decisions, teams can explore more questions earlier:

• Which of these ten concepts is worth taking into human fieldwork?
• Which message is likely to split our audience?
• Which segment reacts with enthusiasm, indifference or concern?
• What objections should we expect before launch?
• Which assumptions in the strategy are weakest?
• Where is the source data strong enough for inference, and where do we need new human evidence?

The output is not a replacement for every survey, focus group or interview.

It is a faster evidence layer that helps teams spend their human research budget better.

The most valuable human research will still be done with humans. Synthetic research helps decide what to ask, who to ask, and which ideas are worth the cost of asking.

What we deliberately do not claim

Synthetic research is a rapidly developing field. The evidence is promising in some settings and cautionary in others.

Our methodology is designed around that reality.

We do not claim:

• that a respondent model is the original human
• that a synthetic response is something the original respondent actually said
• that synthetic respondents are universally interchangeable with human samples
• that increasing “synthetic sample size” creates more independent human evidence
• that an LLM-only counterfactual establishes causality
• that every research question can be reliably inferred from every dataset
• that a fluent qualitative answer is proof of validity

Synthetic research is most useful when it is grounded, bounded, validated and transparent about uncertainty.

For low-researchability questions, novel categories with little relevant source evidence, or decisions requiring direct measurement of current human experience, we recommend human fieldwork.

The standard we are building for

The first generation of AI research tools made synthetic personas easy to create.

The next generation needs to make synthetic research credible.

That means answering harder questions:

• What real human evidence grounded this synthetic response?
• How uncertain was the inference?
• Does the result preserve the source population?
• How does the system perform on answers it was not allowed to see?
• Does it work equally well across research domains and respondent groups?
• Is this new question actually supported by the source data?
• Did a model or prompt update change the result?

Those are not interface details. They are the foundation of trust.

Our approach is to make them part of the product.

Real respondents provide the empirical structure. Question-relevant evidence grounds each inference. Probability preserves uncertainty. Survey weights preserve the source population. Held-out human answers test the system. And when the evidence is weak, the platform says so.

That is our standard for synthetic research.

Research informing our approach

Our methodology is informed by a growing research literature, including:

1. Park, J. S., Zou, C. Q., Shaw, A., Hill, B. M., Cai, C., Morris, M. R., Willer, R., Liang, P., & Bernstein, M. S. (2024). Generative Agent Simulations of 1,000 People. arXiv:2411.10109.
2. Kinzinger, L., & Hartmann, J. (2026). Synthetic Personalities: How Well Can LLMs Mimic Individual Respondents Using Socio-Economic Microdata? arXiv:2606.04592.
3. When Can Digital Personas Reliably Approximate Human Survey Responses? A Multi-Level Evaluation of Predictive Fidelity. (2026). arXiv:2605.10659.
4. Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M. Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis.
5. Ball, S., Allmendinger, S., Kreuter, F., & Kühl, N. (2025). Human Preferences in Large Language Model Latent Space: A Technical Analysis on the Reliability of Synthetic Data in Voting Outcome Prediction. arXiv:2502.16280.
6. Rupprecht, J., Ahnert, G., & Strohmaier, M. (2025). Prompt Perturbations Reveal Human-Like Biases in Large Language Model Survey Responses. arXiv:2507.07188.

Methodology note

This document describes the architecture and validation standard we are implementing. Dataset-specific fidelity, calibration and researchability metrics should be reported from completed validation runs and should not be inferred from results reported in third-party studies.