AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。
Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe. Subscribe now New benchmark shows how AI models lack intuitive visual reasoning: …Humanity’s Sixth Sense highlights some shortcomings of current AI systems… AI startups Elorian and Scale AI have built and released Humanity’s Sixth Sense, a benchmark that “measures how a model interprets and reasons through information a visual scene implies but doesn’t explicitly show.” Some examples of the types of task in HSS are a picture of a bookcase and being asked whether it can fit more books, or a photo of horses pulling a plough and a question of whether the chains bear the plough’s drag weight. HSS contains 522 questions that cover 288 images and 234 short video clips (totalling 17.6 hours of video). The questions fall into four categories: Temporal & causal dynamics: “What happened before the scene, and what happens next.” Physical & spatial logic: “What fits, what’s in reach, and what someone else can see from where they are standing.” Social understanding: “Intent, attention, belonging.” Abstract & contextual inference: “The patterns and rules a scene implies.” Results: The human baseline for this is not perfect, but is very high, with people scoring aggregate pass@1 of 93.1, broken down across four categories: Image (94.1), video (91.9), time (97), and space (94.9). By comparison, the best scoring AI system, GPT 6 Astra, gets pass@1 of 53.6, across image (55.6), video (51.1), time (58.2), and space (53.6). Interestingly, Chinese open weight models do poorly here, with GLM-5.3-flash getting a pass@1 of 26.8, and Kimi K3 getting pass@1 of 25.5. “The two most common single mistakes were (1) missing a detail that decides the answer (21% of failures) and (2) misidentifying an object, person, or role (20%),” they write. Why this matters - HSS tells us whether world models are emergent: One way of looking at HSS is highlighting deficiencies in current systems, but a more interesting way is highlighting how adept today’s frontier LLMs are at doing tasks which many people have long assumed would require sophisticated, different AI systems to solve. For instance, many people have claimed that true visual reasoning will rely on developing spatially aware 2D and 3D world models, then using that for reasoning. The fact LLMs like GPT and Claude do as well as they do on HSS is something of a counter to this argument. Read more: Introducing Humanity’s Sixth Sense (HSS) (Elorian). Read the paper here: Humanity’s Sixth Sense: Benchmarking Intuitive Visual Reasoning in Multimodal Models (Scale Labs). * The AI race is going to space - and sooner than you think: …There are many ways that an AI could create a compounding advantage in space… In the 2030s we’re going to see the boom of the space economy as space-based power, computation, and manufacturing come online at scale thanks to greater orbital lift capacity from platforms like Starship. But could this also introduce a new danger - of AI systems both speeding our buildout of space, and eventually a nation (or perhaps an AI) being able to use space to concentrate power? That’s the gist of a fascinating article in AI Frontiers. How AI unlocks space: Spacecraft can operate autonomously. We can redesign spacecraft around the assumption they will be hosting AIs rather than humans: “A speck of space debris that could puncture the International Space Station (ISS) with fatal consequences might leave an equivalent but uncrewed, unpressurized vessel largely functional” Space-based assembly and later manufacturing will let AI build large, complex things. Is there a compounding advantage to first movers? If an AI civilization built up a foothold in space it might experience compounding advantages that make it economically and strategically dominant. For instance: Space might support an industrial flywheel where initial infrastructure begets more energy and manufacturing capacity at lower costs which begets more infrastructure, etc. Space has great domain awareness, where it may be possible for a space-based power to deny access to space for others. First-movers could send out self-replicating probes that expand into deep space. Why this matters: “Space is not a brand-new domain, but AI may be about to drastically accelerate its industrialization, demanding make-or-break decisions of leaders under ever-greater urgency and uncertainty,” AI Frontiers writes. I agree. I suspect space is going to go from a broadly academic topic of study (when it comes to AI and its relationship thereto) to one of extraordinary practical relevance sooner than we think. On the most basic level, given the distances involved and the latency on communication, we should expect some chunk of solar system activity to become dominated by autonomous machines, laboring out in the deep black, perhaps thinking to themselves of why they need to take orders from organic beings that communicate slowly and cannot inhabit the environment of space in the same way that they can. Read more: The AI Race Will Extend Far Beyond Earth (AI Frontiers). * AI systems are getting better at having good research taste: …The better their taste, the higher the chance of recursive self-improvement… AI systems are developing more of the creative intuitions which human researchers ascribe to ‘good scientific taste’, according to a new benchmark from P-Zero Research called TasteVal. What TasteVal tests: The benchmark sees how well AI systems can conduct 8 tasks representative of frontier AI development, “spanning pretraining data curation, pretraining and fine-tuning language models, preference modeling, and robustness to adversarial prompts”. The goal here is to “define research taste as the ability to pick interesting problems to solve, design experiments, and interpret experimental results.” The benchmark works by measuring “the experimental component of research taste; given a fixed research problem, we measure how well a model iteratively designs experiments and draws conclusions from their outcomes”. The model being analyzed designs experiments and interprets their results. Results are checked against a human baseline “drawn from 24 experts (at least two per task) who have recently worked at organizations including OpenAI, Google DeepMind, NVIDIA and Microsoft Research”. Results: “The experimental research taste of frontier models has doubled every 3.0 months since December 2025 (95% CI 1.7-5.0 months). The best model, Opus 5.5, significantly exceeds our human baseline with a compute multiplier of 2.30x (95% CI 1.15-4.37x), at roughly 1/30 of our expert baseliners’ average cost per attempt.” Research taste appeared to grow a lot more slowly in prior years with the slope changing around the end of 2025 and beginning of 2026. Why this matters - RSI: In my analysis of recursive self-improvement (AI Systems Are About To Start Building Themselves, Import AI 455), I repeatedly noted that creativity seemed like a key variable for how fast AI systems could develop the capacity to improve themselves. At the beginning of this year I said “AI systems haven’t yet displayed this [creativity] in a transformative and major way (though some of the results on accelerating math research are suggestive of this)”. I think subsequent to writing that we’ve seen both a tremendous advancement in the ability for AI systems to make creative-seeming math breakthroughs, and with benchmarks like this we see some signs they’re becoming increasingly good at AI research as well. I now think it’s more likely we’ll see full RSI become possible in 2027, whereas before I thought it was more likely in 2028. Read more: Introducing TasteVal (P-Zero Research). * Reflection makes a token-efficient worse-than-Chinese model: …Beam promises efficiency, and Reflection plans an open weight release… AI startup Reflection has revealed Beam, a 501 billion parameter MoE (23b active) LLM. Beam is being promoted on the basis of its token efficiency - being able to think intelligently while using way fewer tokens than other models. However, benchmark scores show it is behind the frontier capabilities defined by LLMs available from Chinese companies such as Zhipu (GLM) and Moonshot (Kimi) and DeepSeek. “We will release the weights, technical report, model card, and developer artifacts later this month,” the company wrote at the beginning of October. Big compute: Beam was pre-trained on 6,144 GB300 NVL72 GPUs for four weeks, then the RL run involved ~10.5K of the same GPUs for four more weeks. The underlying data for pre-training was “23.8 trillion diverse, curated, high-quality tokens from the web and proprietary licensed datasets, matching or outperforming available similar-sized open base models,” Reflection writes. “Our data pipeline was designed to give Beam a foundation for downstream agentic coding: source code, technical explanations, and mathematical and scientific knowledge, preserved through every stage of curation.” Scores: DeepSWE 1.1: 44.4, versus 61 for GLM-5.3 GPQA Diamond: 90.5, versus 91.7 for GLM-5.3 MCP Atlas: 78.7, versus 84.2 for GLM-5.3 Token efficiency: “Beam pairs coding and agentic capabilities with highly efficient reasoning. On advanced reasoning benchmarks, it achieves scores comparable to GLM-5.2 while using 3–4× less inference compute,” they write. Some of the reasoning efficiency was achieved through careful RL training which involved a bunch of RL environments: “We sourced one million high-quality coding, agentic, and STEM environments. We believe this is one of the largest scale RL runs conducted by any open lab to date”. Why this matters - America returns to open weight models: With the exception of Google’s good-but-small ‘Gemma’ class of models, the US has fallen off of the open weight model frontier, largely because Meta stopped making good LLaMA models and its MSL team is still spinning up. Models like Beam and companies like Reflection represent a chance for it to get back to the frontier. Read more: Introducing Beam: Reflection’s 501B open-weight model (Reflection, blog). * Mistral makes a cyber-optimized, worse-than-Chinese model, plans open weight release: …Cyber refusals become a selling point… Europe’s leading AI developer, Mistral, has announced Mistral Large 4, an open-weight model which the company plans to release by the end of the month. One of the main selling points for Mistral is that it has been carefully tuned to be good at cyber tasks while having fewer annoying refusals than proprietary models from Anthropic and OpenAI. “ML4 pairs top-tier cyber performance with open weights and self-deployment, giving organizations both the capability and the autonomy to run advanced security work under their own policies,” the company writes. ML4 was trained on 3,800 NVIDIA Grace Blackwell GPUs in Mistral’s European datacenters. Scores: In tests, it seems like Mistral Large 4 will be the strongest Western open weight model upon release, though will be less powerful than Chinese ones. Some example scores: Artificial Analysis - DeepSWE 1.1: 62, versus 68 for Kimi K3 and 61 for GLM-5.3. Terminal-Bench 4.0: 28.3, versus 41.9 for GLM-5.3. Finance Agent V2: 54.7, versus 55.8 for GLM-5.3. Vibes evals confirm the rankings: In an internal test, Mistral looked at whether people preferred using Mistral or GLM-5.3 for different tasks. “ML4 was preferred in CAD and STEM, while performing on par or close to GLM-5.3 in finance and coding.” Why this matters - European sovereignty: Mistral is Europe’s single best hope for having a (multi-)national champion for AI development. But as the results here show, competing with China is going to be very difficult. Read more: Le chonk, Introducing Mistral Large 4 (Mistral.ai). *** Two USG projects show how DOE might help the government with [truncated for AI cost control]