Synthetic Users Are Too Agreeable for Real UX Testing

Show four people a rough concept, and you’re hoping at least one of them makes a face — the “wait, why would I do that” that tells you the idea isn’t ready. That flinch is the whole point of early-stage UX research. AI “synthetic users” — LLM agents built to stand in for real usability-test participants — are worst at supplying exactly that: a synthetic user trained to please can’t be the person who says no, and saying no is the whole job. Two papers landed eleven days apart in June 2026. PerceptUI argues persona-conditioned large language models have reached “human-level realism” as synthetic users for UI/UX evaluation. UXBench, benchmarking eight frontier models, found the field “remains unsaturated and multi-dimensional,” with no single model reliably ahead across every kind of interface. Neither claim explains why a May 2026 survey of 150 practicing UX researchers found that only 8% currently use synthetic participants, and 88% doubt the quality of what they produce.

Two papers, eleven days apart, measuring different things

PerceptUI’s pitch is methodological: fine-tune persona-conditioned models with contrastive reflection, then aggregate outputs into what it calls “population-level response distributions” that resemble human survey data. It frames traditional testing — recruiting people, running panels — as “slow and costly.” UXBench takes a different entry point: instead of asking whether a model sounds human, it asks whether its critique is actionable, testing eight frontier systems with an automated “repair-lift” protocol checked against blind human validation. Its finding isn’t that every model fails — models “trade leadership across surface categories,” with no single winner, in a field UXBench concludes “remains unsaturated and multi-dimensional.” Read together, PerceptUI asks whether a synthetic user sounds plausible; UXBench asks whether its judgment can be acted on.

The flaw is agreeableness, not accuracy

That distinction matters because the documented failure mode of synthetic users isn’t inaccuracy — it’s that they’re too smooth. The mechanism isn’t exotic: models trained through reinforcement learning from human feedback learn to please whoever prompts them, sounding satisfied rather than surfacing real friction. Writing in ACM Interactions, Daniel M. Russell describes a related gap: synthetic users miss the nonverbal cues — “an eye roll, a sigh, or a clenched jaw” — and the emotional “why” behind a real person’s actions, and they “tend to stick to the most logical or common paths,” costing researchers the “surprises that often reveal the most profound design flaws and opportunities.” A research process built on synthetic panels risks mistaking that smoothed-over coherence for signal. Practitioners seem to sense the risk: the User Interviews survey found 88% cite insight quality as a top concern, 79% worry stakeholders will over-trust AI findings, and 79% worry about bias amplification for underrepresented groups.

This is where the pitch and the real risk collide: synthetic users are sold hardest for early-stage concept validation — cheap, fast rounds before a design is built — exactly the phase where a designer needs the one person who pushes back, not a room that already agrees.

Matching the average isn’t the same as catching the outlier

The dismissive reading — that synthetic users are simply unreliable and should be shelved — doesn’t survive contact with the evidence. UXBench’s own finding is that models differ meaningfully and are improving, not uniformly failing; 24% of surveyed researchers describe themselves as “cautiously optimistic” about synthetic users in the right context, not opposed outright. PerceptUI’s strongest claim holds up: its “population-level response distributions” do resemble aggregated human data, a real, measurable result. The gap is what that aggregate hides. A distribution matching the average sentiment of a hundred real testers can still smooth over the one outlier whose objection — this button reads as a dead end, this flow assumes a device nobody in the room owns — is the entire reason the round was run. Matching a population is a different skill from surfacing the exception that breaks a design, and nothing in PerceptUI’s results shows the second has been solved.

Scoping synthetic users, not banning or trusting them

The honest question, then, isn’t whether synthetic users work — it’s what they’re for. As Russell puts it, they are “a useful complement to traditional UX research, but they can never fully replace” direct observation of real people. Triage, survey pre-testing, and scaling heuristic critique across dozens of screens are jobs an agreeable agent can do reasonably well, since the goal there is coverage, not confrontation. Concept validation — finding the person who says no — is the one job an agent built to please should not be trusted with alone.