AI-generated survey respondents are cheap, fast and getting popular. The 2026 evidence on how accurate they are is mixed, and one major study was blunt about the limits.
Quick answer: Synthetic respondents (also called silicon samples) can capture broad trends and rank options reasonably well, but they are not accurate enough to measure exact percentages. In a September 30, 2026 study, Pew Research Center found AI-generated survey answers differed from real human results by an average of 12 percentage points across nearly 300 questions. Researchers generally recommend using synthetic data for early exploration and screening, then confirming with real people.
Test your idea with Crowd Before
Published October 15, 2026. Findings are attributed to the linked sources.
Key takeaways
- Pew's 2026 study is the most thorough public test so far. AI "digital twins" of real panelists answered real surveys, and the results often missed by a wide margin.
- Errors were larger for some groups. Pew reported poorer results for Republicans and Republican leaners, Black adults, people without a college degree and infrequent internet users.
- AI respondents tend to be too agreeable and too uniform. Several studies find positive bias and less variation than real people.
- Broad trends still show up. A 2026 NIM study found synthetic answers beat chance and captured broad trends, with best fit for early-stage, lower-stakes use.
- Use rankings and patterns, not exact percentages. Do not treat a synthetic number as market share.
What are synthetic respondents?
Synthetic respondents are AI-generated stand-ins for survey participants. A language model is given a profile (age, income, job, location, habits) and asked to answer questions as that person would. Collect enough of them and you have a synthetic sample, sometimes called a silicon sample or a set of digital twins.
Market research firms, startups and tools use them to test concepts, messages and prices quickly. For background on the industry boom, read AI customer simulation in 2026.
The big test: Pew Research Center, September 30, 2026
Pew built an AI "twin" for each member of its American Trends Panel, using demographic and political typology data. The twins then answered real Pew surveys across three waves, and Pew compared the results with what the real people said.
What Pew reported:
- Average error of 12 percentage points. Across nearly 300 questions, AI estimates differed from human results by 12 points on average. By wave, the average error ranged from 11 to 15 points.
- Worse for some subgroups. The synthetic sample did especially poorly for Republicans and Republican leaners, Black adults, adults who have not attended college and people who rarely use the internet.
- Big misses on specific questions. In one example, 97% of simulated Hispanic respondents said they were likely to follow the World Cup, versus 42% of real Hispanic adults. In another, 72% of simulated Hispanic respondents rated Spanish instruction in schools as extremely important, versus 32% of real ones.
- Gaps in the answers. AI respondents often avoided certain answer options altogether, and different AI models gave different pictures of public opinion.
- Pew's conclusion: AI polling is not a replacement for surveying real humans.
An independent analysis of Pew's published data by FlashInsight adds detail, including that the AI twins chose "not sure" far less often than real people (4% versus 16%). It also notes that ranking among options held up better than exact percentages. That is a secondary analysis, so check Pew's own report for the original figures.
What other 2026 research says
A mixed result from NIM Marketing Intelligence Review
A 2026 study by Kaiser and colleagues tested synthetic respondents in a realistic marketing funnel scenario. It found synthetic responses predicted individual answers significantly better than chance and captured broad trends. The authors concluded the data may currently suit early-stage concept testing or lower-stakes applications, not decisions that need precise insights.
Positive bias in taste and preference
A June 2026 arXiv paper tested AI survey surrogates against real arts-participation data and found a systematic positive bias for liking. Real taste structures were distorted, with some associations flattened and others exaggerated. This matters for product testing, because an "agreeable" respondent can overstate interest.
Homogenization
A 2025 arXiv study found severe homogenization in AI responses compared with human data, and structural inconsistencies in how results held up across different levels of analysis. Earlier work by Bisbee and colleagues in Political Analysis (2024) found synthetic data had less variation than real data and was not reliable for statistical inference, even where averages looked close.
Vendor benchmarks
A CleverX guide reports vendor benchmarks of roughly 85 to 95% alignment on calibrated quantitative trends, falling to about 37 to 60% in replication studies of complex multi-factor questions. Those figures come from vendor claims and secondary summaries, so treat them as a rough range.
Why do synthetic respondents miss?
- They average. Language models tend to give the most likely answer, which erases the spread of real opinion.
- They are too agreeable. Models are trained to be helpful, which can inflate "liking" and "would buy."
- They know too much. They can answer factual questions better than real people.
- They stereotype. Pew noted AI responses could lean on racial or partisan stereotypes.
- Models differ. Different models give different answers to the same questions.
- They look backward. Training data reflects what is already known, so new products and fresh behavior are harder.
What does this mean for startup founders?
Pew tested public opinion polling. Founders mostly use crowds for a different job: screening ideas and finding weak spots. The evidence suggests that job is more reasonable than measuring a population, but only if you use the results carefully.
Do:
- Use synthetic results to compare ideas, prices and segments against each other.
- Look at where buyers drop off and which objections come up.
- Treat strong segments as hypotheses to test.
- Confirm with real customer conversations and a paid demand test.
Do not:
- Read a percentage as market share or a forecast.
- Treat "would pay" as real payment.
- Make a large, hard-to-reverse bet on synthetic results alone.
- Trust any tool that promises a probability of success.
How Crowd Before handles these limits
Crowd Before puts your idea in front of 100 to 5,000 AI-generated buyers shaped by real demographic, income and industry data. Each buyer reacts on their own, and the numbers in your report are counted from those individual reactions rather than written by AI. Every report says plainly that the crowd is AI-generated.
You get a verdict (Build it, Rethink it, or Don't build this yet), a drop-off funnel, who to sell to first, a price to test, and real competitors found on the web. Watch a sample run or read how it works.
No AI crowd has solved the problems above. Positive bias and uniformity are known risks for the whole category, so use Crowd Before as a fast, cheap first test that tells you what to check with real people. It does not prove demand. Your first idea is free, and paid plans start at $4.99.
Run a first test, then confirm with real buyers
A safe workflow for using synthetic data
- Screen several ideas with a crowd test.
- Pick the idea and segment that look strongest, and note the main objections.
- Interview five to ten real buyers using those objections as questions.
- Run a demand test with a price: pre-order, deposit or paid pilot.
- Decide using real behavior as the final word.
See how to test a business idea before building for the full process.
FAQ
What are synthetic respondents?
AI-generated stand-ins for survey participants. A model answers questions as if it were a person with a given profile.
How accurate are synthetic respondents?
It depends on the question. Pew's 2026 test found an average 12-point gap from human results across nearly 300 questions. Other studies find they capture broad trends but under-represent variation and lean positive.
What is a silicon sample?
A set of AI-generated respondents used in place of a real survey sample. Digital twins are a related approach that models specific real people.
Can synthetic respondents replace real surveys?
Not on current evidence. Pew concluded that AI polling is not a replacement for rigorously surveying real humans. Researchers recommend using synthetic data to explore and screen, then confirming with real people.
Are synthetic respondents useful for startups?
They can be useful for early screening, such as comparing ideas, finding weak points and choosing who to talk to first. They should not be the only evidence behind a major decision.
Why are AI survey answers so uniform?
Models tend to give the most likely answer, which reduces the spread you see in real populations.
Can AI respondents be biased?
Yes. Studies report positive bias, stereotyping and different results across models.
How should I use a synthetic result?
Use rankings and patterns, not exact percentages, and confirm with real customers before committing money.
Sources
- Pew Research Center, Can AI Stand In for Human Survey-Takers? (Sept. 30, 2026)
- FlashInsight, analysis of Pew's data
- NIM Marketing Intelligence Review, Leaving Insight to Digital Twins? (2026)
- Ma et al., Not-quite-human tastes (arXiv, June 2026)
- Li, Li and Qiu, ChatGPT is not A Man but Das Man (arXiv, 2025)
- CleverX, synthetic respondents vs real participants (2026)
Related reading: can ChatGPT validate your business idea? Questions about using AI in your research? Ask in the founder community.
Last updated: October 15, 2026.