Fei-Fei Li Strikes Again: The ImageNet of Spatial Intelligence Arrives

Fei-Fei Li, a renowned AI researcher known for spearheading the ImageNet project, has once again made a significant move in the field of artificial intelligence. Her team at Stanford University, along with collaborators from UCLA, Tsinghua University, and Northwestern University, recently introduced ESI-Bench—a new benchmark designed to evaluate embodied spatial intelligence.

Unlike previous benchmarks that passively present optimal viewpoints to models, ESI-Bench transforms the observer into an active agent. It closes the perception-action loop by forcing AI agents to take actions—such as moving around, picking up objects, or pouring liquids—to gather the information needed to answer spatial questions. This marks a fundamental shift from testing "vision" to testing true spatial reasoning.

ESI-Bench encompasses 10 task categories, 29 subcategories, and 3,081 task instances built on the OmniGibson simulation platform using the BEHAVIOR-1K scene library. The tasks are grounded in Spelke's four core knowledge systems: object representation, layout and geometry, numerical representation, and goal-directed action. A key feature is "action forcing": each problem requires the agent to actively perform actions to obtain sufficient information. For example, in a "rigid containment" task, the agent must approach, bend down, and even lift containers to inspect their true capacity from different angles.

The team conducted comprehensive tests on state-of-the-art multimodal large models, including GPT-5 and Gemini series. Three key findings emerged:

First, perception is not the bottleneck—action is. While active exploration improved performance (e.g., Gemini 3.1's accuracy on partially occluded tasks jumped from 14.6% to 95.1% when given optimal viewpoints), the models struggled to discover those viewpoints on their own. Worse, random multi-view strategies actually hurt performance: GPT-5's accuracy on spatial distance tasks dropped from 53.9% to 49.1% when given multiple random images. The team termed this "action blindness," where a poor action leads to a poor viewpoint, triggering an irreversible cascade of failures.

Second, imperfect 3D reconstruction can be worse than 2D. When given ground-truth 3D data, models performed well—Gemini's accuracy on material transparency tasks improved from 44.0% (2D) to 60.4% (3D). However, when using real-world 3D reconstructions from the VGGT model, geometric configuration task accuracy plummeted from 27.5% (2D baseline) to just 9.9%. The distortions and errors in reconstructed 3D scenes provided "poisonous" inputs that misled reasoning.

Third, models exhibit a metacognition deficit: they don't know when they have gathered enough evidence. While humans actively seek disconfirming evidence and adjust confidence under ambiguity, models stop exploring prematurely and make overconfident judgments. For example, in physical contact tasks, humans achieved 88.3% accuracy while GPT-5 managed only 64.2%. This fundamental flaw cannot be solved by better vision encoders or more exploration steps alone.

The paper's first author is Yining Hong, a postdoc at Stanford advised by Yejin Choi and closely guided by Leonidas Guibas, Jiajun Wu, and Fei-Fei Li. The team also includes Jiageng Liu (UCLA PhD student), Han Yin (Tsinghua undergrad), and professors Fei-Fei Li, Jiajun Wu, Yejin Choi, Manling Li, and Leonidas Guibas.

ESI-Bench is available on arXiv and the project website. It provides a systematic framework for evaluating embodied spatial intelligence, highlighting that while AI can "see" well, it is still far from possessing spatial intelligence that can move, touch, and actively seek answers.