Moonshot AI's PerceptionBench: No frontier model reaches 60% on visual perception
Moonshot AI's new PerceptionBench isolates visual perception from reasoning, testing ten atomic sub-skills with questions that require only looking at an image. No frontier model surpasses 60 percent accuracy; GPT-5.6 Sol leads at 59.7 percent, narrowly ahead of Kimi K3, with open-source models far behind. The benchmark reveals that hallucination is the weakest skill, with top models scoring below 27 percent at inventing objects that don't exist. The authors argue many apparent reasoning failures actually originate in perception, suggesting that improving multimodal models' ability to read images could be a key bottleneck. The dataset and evaluation code are publicly available on GitHub, allowing broader testing and development. However, the benchmark is from a single organization, so independent validation is needed.
Moonshot AI's PerceptionBench: No frontier model reaches 60% on visual perception
Moonshot AI's PerceptionBench tests how well multimodal AI models can actually "see," separate from logical reasoning. No frontier model reaches 60 percent accuracy, and GPT-5.6 Sol leads by a narrow margin. PerceptionBench breaks vision down into ten atomic sub-skills instead of lumping perception, knowledge, and reasoning into a single task.
Key takeaway
Multimodal models' visual perception is a significant bottleneck, with even leading models like GPT-5.6 Sol scoring under 60% on PerceptionBench, particularly struggling with hallucination.
What happened
Moonshot AI, the company behind the Kimi assistant, has released PerceptionBench, a new benchmark designed to isolate visual perception from reasoning. The benchmark breaks vision into ten atomic sub-skills, such as counting, attributes, and hallucination, with questions that can be answered purely by looking at an image. According to The Decoder, no frontier model reaches 60 percent accuracy, with GPT-5.6 Sol leading at 59.7 percent, followed closely by Kimi K3 at 58.5 percent and Claude Fable 5 at 57.2 percent.
The results highlight hallucination as a major weakness, where GPT-5.6 Sol scores only 26.9 percent despite overall leadership. The authors argue that many supposed reasoning errors actually occur at the perception stage, and that existing benchmarks each capture only a narrow slice of perception errors. The dataset and evaluation code have been made available on GitHub, and Moonshot AI also references related benchmarks like WorldVQA and BabyVision that similarly show poor visual perception performance.
Evidence
No frontier model reaches 60 percent accuracy on PerceptionBench; GPT-5.6 Sol leads at 59.7 percent.
The Decoder · attributed
No frontier model reaches 60 percent accuracy, and GPT-5.6 Sol leads by a narrow margin.
PerceptionBench breaks vision into ten atomic sub-skills, each answerable by looking at the image alone.
The Decoder · attributed
PerceptionBench breaks vision down into ten atomic sub-skills instead of lumping perception, knowledge, and reasoning into a single task.
Hallucination is the weakest skill; GPT-5.6 Sol scores only 26.9 percent there.
The Decoder · attributed
On average, "hallucination" is the weakest skill across the board. GPT-5.6 Sol scores only 26.9 percent there.
The authors argue that many reasoning errors actually happen at the perception level.
The Decoder · attributed
The authors argue that many multimodal model failures typically chalked up to "reasoning errors" actually happen at the perception level.
Why it matters
This benchmark could redirect AI development toward perception skills, as many reasoning errors may actually stem from misreading images, affecting reliability in real-world applications.
Limits and uncertainties
The benchmark is from a single organization, Moonshot AI, and may not represent all visual perception aspects.
Results are based on the reported accuracy numbers, which have not been independently verified.
Practical implications
Developers should use perception-specific benchmarks like PerceptionBench to identify granular weaknesses in their multimodal models.
Improving perception, especially reducing hallucination, could be a key area for model improvement.
What to watch
Updates from Moonshot AI or independent evaluations on PerceptionBench scores.
Release of similar benchmarks from other organizations to validate findings.
