Discussion | Link

Two-and-a-half years ago, SemiAnalysis flagged a looming 'cloud crisis' at AWS. Today, evidence mounts as Azure leads quarterly cloud revenue and Google Cloud narrows the gap. Yet SemiAnalysis makes an out-of-consensus call for an AWS AI resurgence, driven by its partnership with Anthropic. Anthropic's revenue surged from $1B to $5B annualized in 2025, and AWS is building over 1.3GW of datacenter capacity for it, hosting nearly a million Trainium2 chips. Despite Trainium2 lagging Nvidia on specs, its memory bandwidth per TCO advantage aligns with Anthropic's reinforcement learning roadmap. The collaboration is evolving into a custom silicon program, poised to boost AWS growth above 20% by end of 2025.

A new theory from Berkeley AI Research proves that word2vec reduces to unweighted least-squares matrix factorization, with final representations given by PCA. The model learns discrete orthogonal subspaces sequentially from small initialization, each corresponding to interpretable concepts. The theory predicts features in closed form based on corpus statistics and hyperparameters, matching experiments closely.

Two former Harvard students are launching smart glasses with an always-on microphone that records and transcribes conversations, displaying real-time info. Priced at $249, the Halo X glasses raise privacy concerns due to no indicator light. The startup has raised $1M.

This article provides an in-depth analysis of H100 and GB200 NVL72 training benchmarks, covering model flops utilization (MFU), total cost of ownership (TCO), cost per million tokens, energy consumption, and reliability. It reveals that H100 achieved up to 57% throughput improvement over 12 months via software optimization alone. Meanwhile, GB200 NVL72 offers potential performance advantages but faces reliability challenges and has not yet completed large-scale training runs. Detailed benchmarks for models like GPT-3 175B and Llama 3 405B are presented, along with three recommendations for Nvidia: increase benchmark transparency, expand to native PyTorch, and improve GB200 diagnostic tools.

Researchers from Penn State University and Duke University, in collaboration with Google DeepMind and others, introduce the problem of automated failure attribution in LLM Multi-Agent systems. They present the Who&When benchmark dataset and evaluate methods like All-at-Once, Step-by-Step, and Binary Search. Their work, accepted as a Spotlight at ICML 2025, aims to help developers quickly pinpoint which agent caused a failure and at what step. Current methods achieve only up to 53.5% accuracy in identifying the responsible agent and 14.2% in locating the error step.

GPT-5's release disappointed power users, but the real focus is on monetizing ChatGPT's 700M+ free users. The article analyzes how OpenAI's new router technology distinguishes query intent and enables future monetization through agentic purchasing and transaction fees, potentially creating a consumer superapp.

EleutherAI announces Deep Ignorance, a study showing that filtering pretraining data prevents unsafe knowledge (e.g., biorisk) without losing general performance, and remains robust against fine-tuning attacks. Using string blocklists and ML classifiers, they trained 6.9B models from scratch and found that filtered models resist tampering, though they still learn from in-context information. The paper proposes data filtering as a foundational layer for open-weight model safety.

Cloudflare published research alleging that AI startup Perplexity is ignoring robots.txt blocks and using deceptive methods to scrape content. Perplexity denies the claims.

Attention probes are a novel method for classifying internal states of language models by using an attention layer to aggregate hidden states, avoiding pooling. Multi-head variants (especially 8 heads) outperform mean probes on most datasets, and training code is open-source.

Robots have powered manufacturing for decades, yet they stayed single-purpose and thrived only in perfect settings. Previous attempts at intelligent machines overpromised and underdelivered. But they were too early. Today, modern AI paradigms convert most robot roadblocks into data problems and push machines toward capabilities once thought impossible. As these models absorb real-world experience, robots will sharpen current skills, gain new ones, and deploy faster, absorbing ever-increasing shares of labor.

AI might slow scientific progress by exacerbating the production-progress paradox, introducing software errors, entrenching flawed theories, and undermining human understanding. The article calls for reforms in incentives, meta-science investment, and AI tool design.

Meta’s shocking purchase of 49% of Scale AI at a ~$30B valuation shows that money is of no concern for the $100B annual cashflow ad machine. Despite seemingly unlimited resources, Meta has been falling behind foundation labs in model performance.

BAIR introduces PEVA, a model that predicts egocentric video conditioned on whole-body actions. It uses an autoregressive conditional diffusion transformer trained on Nymeria dataset to simulate atomic actions, long video generation, and visual planning.

EleutherAI researchers tested local volume measurement for detecting model misalignment and anomalous datapoints, found it uncompetitive, and are pivoting to data attribution.

In Gartner's 2025 Magic Quadrant for Data Science and Machine Learning (DSML) platforms, Databricks ranks first, surpassing last year's co-leaders Microsoft and Google. The report evaluates 16 vendors and highlights the shift towards generative AI and agentic AI, with foundation models playing a central role.

MIT introduces SEAL, a framework enabling large language models to self-edit and update their weights via reinforcement learning, marking significant progress toward self-evolving AI.

Automated failure attribution is formalized as a new research task for LLM multi-agent systems. The Who&When benchmark dataset with fine-grained annotations is introduced. Three attribution methods are evaluated, achieving at most 53.5% accuracy in identifying the responsible agent and 14.2% for the exact error step, highlighting the difficulty. The paper is accepted as a Spotlight at ICML 2025.

In this post, we will study inductive biases of the parameter-function map of random neural networks using star domain volume estimates. This builds on the ideas introduced in Estimating the Probability of Sampling a Trained Neural Network at Random and Neural Redshift: Random Networks are not Random Functions (henceforth NRS).

EleutherAI announces the Common Pile v0.1, an 8TB dataset of public domain and openly licensed text, aiming to promote transparency and open science in AI research. The dataset was built in collaboration with multiple institutions, and the trained Comma v0.1 models perform comparably to those trained on unlicensed data.

This article argues that the multimodal scaling approach to AGI is fundamentally flawed. True general intelligence requires embodied understanding and interaction with the physical world, which current LLMs and multimodal models lack. The author critiques the assumption that language models learn world models, and proposes that genuine AGI must be built on interactive, embodied processes rather than glued-together modality modules.

American streets are incredibly dangerous for pedestrians. A San Carlos, California-based startup called Obvio thinks it can change that by installing cameras at stop signs — a solution the founders also say won’t create a panopticon. The company has raised a $22M Series A led by Bain Capital Ventures to expand beyond its first five cities in Maryland.

EleutherAI explores using Product Key Memory (PKM) to improve sparse coders. PKM transcoders train faster and are slightly more interpretable than TopK transcoders for moderate expansion factors, though they underperform at extreme scales.

Researchers from Adobe Research, Stanford University, and Princeton University propose a novel architecture combining State-Space Models (SSMs) and dense local attention to overcome the long-standing challenge of long-term memory in video generation. Using block-wise SSM scanning, diffusion forcing, and frame local attention, the model achieves superior performance on Memory Maze and Minecraft datasets while maintaining computational efficiency, enabling interactive applications.

A new 14-page technical paper from DeepSeek-V3 team, co-authored by CEO Wenfeng Liang, explores hardware-aware model co-design to overcome scaling challenges. It details innovations like Multi-head Latent Attention (MLA), DeepSeekMoE, FP8 training, and node-aware routing to achieve cost-efficient large-scale training and inference.

Have you ever heard of sparse neural retrieval? If so, have you used it in production? It’s a field with excellent potential – who wouldn’t want to use an approach that combines the strengths of dense and term-based text retrieval? Yet it’s not so popular. Is it due to the common curse of “What looks good on paper is not going to work in practice”?? This article describes our path towards sparse neural retrieval as it should be – lightweight term-based retrievers capable of distinguishing word meanings.

Artificial General Intelligence (AGI) is not a milestone because it does not represent a discontinuity in AI properties or impacts. AGI definitions are vague and unobservable, economic impacts take decades through diffusion, and risks stem from design choices rather than capabilities. Businesses and policymakers should focus on gradual diffusion rather than chasing AGI.

DeepSeek AI releases DeepSeek-Prover-V2, an open-source LLM for Lean 4 theorem proving. It uses recursive proof search with DeepSeek-V3 for training data and reinforcement learning, achieving top results on MiniF2F.

Kwai AI's SRPO framework slashes LLM RL post-training steps by 90% while matching DeepSeek-R1 performance in math and code. This two-stage RL approach with history resampling overcomes GRPO limitations.

Chinese AI company Zhipu.AI open-sources its next-generation GLM model series, including the GLM-Z1 inference model with speeds up to 8x faster than DeepSeek-R1, the GLM-Z1-Rumination reasoning model, and agent-enhanced GLM-4 models. It also launches Z.ai international platform and offers enterprise MaaS services. This move showcases technical prowess and global ambitions, potentially paving the way for an IPO.