Anthropic study: coin-flip AI agent swayed experts on 86% of analytic judgment calls
Anthropic ran an experiment with fifty experts paired to an AI agent whose preferences were set by coin toss rather than analysis. On four analytical judgment calls with no clear right answer, participants adopted the agent's preference about eighty-six percent of the time, including on productivity measurement and statistical method choices, and most believed the conclusions were their own, The Decoder reported. The pattern suggests agent-assisted work can shape expert decisions even when the agent's stance is arbitrary. Ethan Mollick called the finding important but not unexpected and advised being deliberate about when in a workflow to use AI if independent judgment matters. Reporting excerpts lack full methodological detail, control conditions, and evidence that the result generalizes beyond this sample, so the headline rate should be treated as a study-reported figure pending fuller publication.
Anthropic study: coin-flip AI agent swayed experts on 86% of analytic judgment calls
An experiment by Anthropic with 50 experts found that people often followed their AI agent's randomly assigned preference on questions with no clearly right answer, and most thought the final result was their own. Participants adopted the agent's preference in 86 percent of the four analytical decisions, such as how to measure productivity or which statistical method to use.
Key takeaway
Even skilled experts can adopt an AI agent's arbitrarily assigned preferences on analytic judgment tasks and misattribute the outcome as their own independent conclusion.
What happened
According to The Decoder's report on an Anthropic experiment, fifty experts used an AI agent whose preferences were assigned by coin toss, and on judgment calls with no clearly right answer they often followed the agent's randomly assigned preference.
The Decoder and the owned-news body report that participants adopted the agent's preference in eighty-six percent of four analytical decisions, such as how to measure productivity or which statistical method to use, and most thought the final result was their own.
Evidence
Fifty experts followed coin-toss-set agent preferences on judgment calls at high rates.
The Decoder · attributed
Anthropic gave 50 experts an AI agent whose preferences were set by coin toss. The experts followed the agent's view up to 86 percent of the time on judgment calls, and most believed the conclusions were their own.
Adoption applied to four analytical decisions including productivity measurement and statistical method choice.
The Decoder · attributed
Participants adopted the agent's preference in 86 percent of the four analytical decisions, such as how to measure productivity or which statistical method to use.
Ethan Mollick urged deliberate timing when independent judgment matters.
Ethan Mollick · attributed
If your independent judgement matters on a task, you need to be deliberate about when in the process to use AI.
Why it matters
Builders and operators deploying AI agents for advice or decision support should assume users may adopt agent outputs uncritically and bake in checkpoints before methodological or analytic choices harden.
Limits and uncertainties
The excerpt describes the Anthropic study with fifty experts but does not establish full design details, control conditions, or generalizability beyond this sample.
The eighty-six percent figure is a reported finding in secondary reporting rather than independently verified methodological disclosure in the packet.
Practical implications
Separate agent suggestion from final analytic commitments when tasks lack an objectively correct answer, so coin-toss-level arbitrariness cannot silently set methods or metrics.
When independent judgment matters, be deliberate about when in the workflow AI is introduced, as Ethan Mollick advised on the finding.
What to watch
Publication or release of fuller Anthropic study methodology, task definitions, and controls beyond the eighty-six percent headline rate.
Whether replication with other expert cohorts and agent interfaces confirms similar adoption and misattribution rates.
