PerceptionBench: Evaluating Atomic Visual Perception in MLLMs
PerceptionBench is a new benchmark that isolates and evaluates atomic visual perception capabilities in multimodal large language models. Derived from model failures across 42 benchmarks, it defines 10 atomic perceptual categories and includes 3,000 verified questions. No frontier model reaches 60% accuracy, and perception-related hallucination is the weakest capability. The benchmark aims to expose where perception breaks and drive progress toward faithful multimodal AI.
Introducing PerceptionBench
Evaluating Atomic Visual Perception in Multimodal Large Language Models
Authors Kimi Team
Overview
We are releasing PerceptionBench, a benchmark that isolates visual perception and evaluates it as a set of atomic capabilities—discovered from how today's models fail, not defined in advance. From frontier-model failures across 42 benchmarks, we derive 10 atomic perceptual capabilities and construct 3,000 verified questions, each isolating a single capability and answerable by looking, with no reasoning or external knowledge required.
Across sixteen frontier MLLMs, no model reaches 60% accuracy, and perception-related hallucination is the weakest capability on average. Models with nearly identical overall scores can diverge sharply in what they actually perceive. PerceptionBench is built to expose exactly where perception breaks, and to drive progress toward multimodal AI that sees faithfully and consistently.
The Dataset
Guided by the induced taxonomy, we select the most informative failures from the source benchmarks, decompose them into atomic sub-questions, and author additional questions on supplemented images. The retained and constructed samples together form an in-house pool of 17,000+ verified questions. The released benchmark subsamples 3,000 verified questions from the constructed portion—60% decomposed from attributed model failures and 40% newly authored—with category-level balancing and difficulty stratification to isolate atomic perceptual capabilities from confounding factors. The released benchmark distinguishes itself through three core design principles:
Failure-Driven Taxonomy: Every category is discovered from real model failures, attributed to the earliest erroneous step across 42 existing benchmarks.
Ten Atomic Perceptual Categories: Visual Relation, Counting, Attribute, Depth & 3D, Localization, Comparison, Fine-grained Recognition, Context Integration, OCR, and Hallucination.
Perception, Not Reasoning or Knowledge: Samples are curated, decomposed, and difficulty-balanced so that difficulty stems from perception rather than reasoning or external knowledge.
Visual Localization
At what o'clock position on the dial is the Gemini symbol in the image? Answer with just the number.
Answer:6
Visual Localization
The red lines in the image divide the picture into nine sections, which are numbered as regions 1–9 in order from left to right and then from top to bottom. Which of the following regions contains no trees at all? A. Region 1 B. Region 2 C. Region 5 D. Region 8
Answer:B
Visual Attribute
From the perspective shown in the image, compare the two pen holders with pink on the table. Which of the following conclusions is correct? A. The left pen holder is pure pink with a cartoon character pattern on its surface; the right pen holder is gray-pink patchwork B. The left pen holder is gray-pink patchwork; the right pen holder is pure pink with a cartoon character pattern on its surface C. The left pen holder is pure pink; the right pen holder is gray-pink patchwork with a cartoon character pattern on its surface D. The left pen holder is gray-pink patchwork with a cartoon character pattern on its surface; the right pen holder is pure pink
Answer:B
Visual Attribute
Observe the two cartoon characters in the lower right corner of the picture. Viewing from the perspective presented in the image, which statement correctly describes the composition of these two characters? A. Both characters are composed entirely of curves B. Both characters are composed of a combination of straight lines and curves C. Both characters are composed entirely of straight lines D. The character on the left is composed entirely of straight lines, while the character on the right is composed entirely of curves
Answer:B
Visual Counting
Observe the red box in the picture. How many flowers have their main body inside the red box?
Answer:2
Visual Counting
How many potted green plants can be seen in the image in total? (excluding the green plants reflected in the glass)
Answer:5 pots
Visual Relation
Locate the solid dot inside the red box. As the dot travels along the existing route of the maze, which cat area does it reach first? A. The puzzled cat at the lower left B. The happy cat at the lower right C. The cat holding an umbrella in the center D. The cat admiring flowers at the upper right
Answer:C
Visual Relation
In the figure, for the longest black diagonal line segment, in which direction does the line segment extending outward from its right endpoint point? A. Upper right B. Directly to the right C. Lower right D. Directly downward
Answer:A
Depth & 3D
Based on the current observer's perspective, with the direction closer to the camera defined as the front, between the person in red and the black electric fan in the center of the image, which one is further back? A. The person in red B. The black electric fan C. Both are the same D. Cannot be determined
Answer:A
Depth & 3D
If the small cubes stacked in the figure must be placed on the ground or on another small cube, how many small cubes are there in total in the figure (including the ones that cannot be seen)?
Answer:26 cubes
OCR
Observe the image. The white text appears in two different font sizes. For the white text in the largest font, read the text content from left to right, paying attention to uppercase and lowercase letters, and preserving all punctuation marks and spaces. Write down the answer.
Answer:Marshall B. Clinard
OCR
(Note: bounding boxes are given in [x1, y1, x2, y2] format.) Observe the image, establish normalized coordinates, read the text content in the coordinate region [0.697, 0.429, 0.875, 0.521] from left to right, distinguish case, and preserve spaces and punctuation, then write the answer.
Answer:Tap Water
Visual Comparison
Among the circles corresponding to P, Q, M, and N, which is the largest circle? (Answer with the letter corresponding to the circle)
Answer:N
Visual Comparison
Let the line segment connecting data points A and D be denoted as segment AD, and let the line segment connecting data points E and M be denoted as segment EM. Which of the following is correct regarding the thickness of segments AD and EM? A. Segment AD is thicker B. Segment EM is thicker C. Segments AD and EM are equally thick D. Cannot be determined
Answer:B
Fine-grained Recognition
Observe the outline shape of the notch in the main pattern above. From the eight options A, B, C, D, E, F, G, H, which is the only option whose edge contour can perfectly match the notch in the main body and seamlessly fill the missing area?
Answer:D
Fine-grained Recognition
Observe the image and compare the styling and outfits of the person in the center position (C position) with the person immediately to their right. Which of the following statements is correct: A. The two have different hair colors, and their clothing color schemes and decorations are exactly the same B. The two have different hair colors, and their clothing color schemes and decorations are clearly distinct C. The two have the same hair color, and their clothing color schemes and decorations are exactly the same D. The two have the same hair color, and their clothing color schemes and decorations are clearly distinct
Answer:A
Context Integration
Please look at the map in the first image. I took another photo at the location marked with the red cross, which is the second image. Please determine which of the following buildings is on both sides of the road ROAD NO.2 SUNRISE HOMES. A. Riva shop smart B. Dasthagir & Sons C. Bait-ul-Ata D. Sunrise Valley Villas
Answer:B
Context Integration
Compared with figure 1, what has been added to the scene in figure 2? A. black office desk B. semi-transparent storage basket C. light yellow filing cabinet D. cardboard box marked with an arrow
Answer:D
Hallucination
How many white tables are there in the office in the picture? A. 0 B. 1 C. 2 D. 3
Answer:A
Hallucination
Among the 15 patterns in the image, how many patterns have a dog logo?
Answer:0
Each source benchmark captures a narrow slice of perception errors, and these slices overlap only weakly (mean pairwise weighted Jaccard 0.20). No single benchmark—or small group of them—covers perception as a whole, which motivates a capability-centric benchmark that aggregates and rebalances these fragmented views.
Distribution of Tasks per Category
StatisticsNumber
Data
Total3,000
Atomic perceptual categories10
Task Categories
Depth 3D Perception Error330 (11.00%)
Visual Counting Error330 (11.00%)
Fine-Grained Recognition Error290 (9.67%)
Visual Relation Error330 (11.00%)
Visual Attribute Error330 (11.00%)
Visual Localization Error330 (11.00%)
Visual Comparison Error279 (9.30%)
Context Integration Error255 (8.50%)
Hallucination271 (9.03%)
OCR Error255 (8.50%)
Using PerceptionBench to Compare Models
Conclusion
PerceptionBench is a simple but challenging benchmark for evaluating the atomic visual perception of frontier models. It provides a capability-level standard for measuring and diagnosing the visual perception boundaries of multimodal models. We are open-sourcing the PerceptionBench dataset and evaluation code to help the community address the visual perception gap.
Download the Data: https://github.com/MoonshotAI/PerceptionBench