Look at a clock face and answer whether the short hand is closer to '4' or '5.' Count how many flowers are in a red box. Determine which of two pen holders is the gray-and-pink combination versus the plain pink one with a character pattern. Tasks that a human could finish in a few seconds—yet more than half of today's most advanced multimodal AI models get them wrong.
A benchmark called "PerceptionBench" (arXiv: 2607.24957), released by Moonshot AI's research team in July 2026, consists of 3,000 such purely "look-only" questions. No reasoning or external knowledge is required at all—correctly reading the image is enough to reach the right answer. And yet, among the 16 models evaluated, the top score was just 59.7%. In other words, even the most advanced AI models still "misread" images more than 40% of the time on pure visual perception.
The "Map Full of Holes" Revealed by 42 Benchmarks
Numerous benchmarks exist for evaluating multimodal large language models (MLLMs), including MMMU and MathVista. But these are designed to measure perception, knowledge, and reasoning bundled together in a single task. When a model gets an answer wrong, there's no way to isolate whether the cause was "misreading the image," "a logical misstep," or "insufficient knowledge."
Moonshot AI's team tackled this problem head-on. They traced back through state-of-the-art model failures across 42 existing benchmarks one by one, identifying the earliest point of failure in each chain of errors. Clustering these results to build an integrated error taxonomy, they found that perception-driven errors converged into 10 categories: visual relationship understanding, counting, attribute recognition, depth and 3D perception, localization, comparison, fine-grained recognition, contextual integration, OCR, and perception-induced hallucination.
There was an even more important finding. The types of errors detected by each of the 42 benchmarks barely overlapped with one another. Each benchmark illuminates only a narrow slice of the space of perceptual failures, and no single benchmark—or even a small combination of them—captures the full picture. The research team notes in the paper that "existing benchmarks are fragmented and rely on heuristic categorizations based on designers' intuitions."
| Perspective | Conventional Benchmarks (MMMU, etc.) | PerceptionBench |
|---|---|---|
| What is measured | Perception + knowledge + reasoning combined | Visual perception isolated and evaluated alone |
| Category design | Predefined by designers (top-down) | Induced from actual model failures (bottom-up) |
| Number of questions | Thousands to over 10,000 (composite tasks) | 3,000 (perception-focused) |
| Perceptual errors covered | Only a narrow slice per benchmark | Systematically covers 10 atomic perceptual abilities |
| Conditions for correct answer | Reasoning and knowledge also required | Answerable just by looking at the image |
The True Identity of "Reasoning Failures" Was "Perception Failures"
There's an even deeper intent behind PerceptionBench's design. When multimodal models fail on multi-step tasks, the common interpretation has been to attribute this to "insufficient reasoning ability." But according to the research team's analysis, when you trace back the chain of failures, the first stumble often occurs at the image-reading stage. If reasoning is built on top of information that was misread during perception, the conclusion will be wrong no matter how correct the subsequent steps are.
To visualize this structure, PerceptionBench decomposes each task into "perception-only sub-questions." Complex multi-step tasks are broken into their constituent elements, and each element is tested to see whether it can be solved through pure perception alone. From an internal pool of over 17,000 questions, 3,000 were extracted with balance across ability categories and stratification by difficulty. Of these, 60% (1,800 questions) were derived by decomposing model failures observed on existing benchmarks, while 40% (1,200 questions) were newly created for supplementary images.
There are also safeguards for evaluation reliability. Answers were scored by GPT-oss-120B judging agreement with reference answers, achieving 99.7% agreement with a human audit of 300 questions. Score stability was also confirmed: across four repeated runs of Kimi K3, scores fell within a range of 58.2% to 58.8%, with a standard deviation of 0.32 points.
The Uneven Capabilities Hidden Behind Similar Overall Scores
The overall scores of the 16 models ranged from 32.5% (GLM-4.6V) to 59.7% (GPT-5.6 Sol). The top five models differ by less than 4 points, which at first glance looks like a tight, evenly matched race.
But the picture changes completely when you look at category-level scores. GPT-5.6 Sol, the overall top scorer, achieves a high 76.7% on Localization, yet sinks to a mere 26.9%—among the lowest—on perception-induced hallucination. Conversely, Gemini 3.5 Flash, which ranks mid-pack overall (52.0%), records the highest hallucination score of all models at 50.6%.
The hallucination sub-test measures a model's ability to answer "zero" when asked about an object that doesn't exist in the image. The research team speculates that models tuned to answer more assertively tend to claim plausible-sounding but nonexistent visual content. Looking at average perception scores, hallucination ranks lowest among the 10 categories at 36.7%, far below visual relationship understanding (53.2%), OCR (52.4%), and localization (52.0%).
| Model | Overall | Localization | Hallucination |
|---|---|---|---|
| GPT-5.6 Sol | 59.7% | 76.7% | 26.9% |
| Kimi K3 | 58.5% | (2nd overall) | (relatively low on hallucination) |
| Claude Fable 5 | 57.2% | ||
| Gemini 3.1 Pro | 56.2% | ||
| Gemini 3.5 Flash | 52.0% | 50.6% (highest) | |
| GLM-4.6V | 32.5% |
Among open-source models, Kimi K3 came within 1.2 points of GPT-5.6 Sol at 58.5%, surpassing every other proprietary model. A correlation between model scale and perceptual performance was also confirmed: Qwen3.5-397B-A17B (47.5%) outperformed GLM-4.6V (32.5%) by 15 points. Even so, not even the strongest models reached 80% in every category.
Accumulating Evidence of What Toddlers Can Do But AI Can't
PerceptionBench is not an isolated achievement. Moonshot AI's team released WorldVQA earlier in 2026. That benchmark specializes in "naming" objects and landmarks—a knowledge-focused set of 3,500 questions—and even the top-performing Gemini 3 Pro reached only 47.4% under conditions that excluded reasoning.
In a separate study, Chinese institutions centered around UniPat AI released BabyVision. It consists of 388 visual tasks solvable even by 3-year-old children (tracing lines, counting hidden blocks, rotating shapes, etc.), and the top score among state-of-the-art models was 49.7% (Gemini 3 Pro). Adult humans scored 94.1%, a gap of 44.4 points. The BabyVision authors call this divergence the "verbalization bottleneck." Their explanation: because current multimodal models convert images into linguistic representations before reasoning about them, geometric information that cannot be faithfully expressed in language (such as the precise curvature of a boundary line or the exact location of an intersection) gets lost in the conversion process.
Within this broader trend, PerceptionBench positions itself as the most fine-grained attempt yet to show "which elements of perception are missing, and to what degree." While WorldVQA asks "what does the model remember," and BabyVision asks "can the model perform toddler-level vision," PerceptionBench asks: "decompose the act of seeing itself into 10 atomic abilities, and measure the achievement level of each one."
Questions That Remain
The research team itself acknowledges certain limitations. First, this taxonomy was induced from the failures of current-generation models, and it will need to be rebuilt as models improve. Second, it hasn't been verified how strongly perception scores correlate with success on real-world composite tasks. The paper explicitly states: "Because PerceptionBench isolates perception by design, capability scores do not directly predict end-to-end task performance." Third, the failure-attribution analysis depends on more powerful analyzer models, meaning that errors in attribution could propagate into the capability labels.
Does this mean we need architectures that preserve visual information without compressing it into language? Or can perception be improved simply by redesigning training data within existing frameworks? The "60% wall" that PerceptionBench has presented is merely a starting point for answering that question.