Token per watt becomes the defining metric as storage moves to AI’s critical path

Token per watt — not raw compute — is emerging as the defining efficiency metric for AI data centers, putting storage at the center of an infrastructure rethink that is reshaping how the industry measures performance, cost and scale.

As agentic AI drives an explosion in context memory demand, the role of solid-state storage has shifted from afterthought to critical path. KV cache offload, inference data pipelining and expanding context windows are converging to create a new storage tier inside AI clusters — one that determines whether GPUs stay busy or sit idle, according to Avi Shetty (pictured), vice president of AI ecosystem, solutions and market enablement at Solidigm, a trademark of SK Hynix NAND Product Solutions Corp.

“The new metric for data centers is token per watt, for dollar per token per watt,” Shetty said. “A single prompt — like, we are in Paris, hey, top five recommendations for restaurants — that simple prompt translates to around 13 gigabytes of data when you account for system activities, tool activities, database requests. When an eviction occurs, you need to rebuild the cache. When you rebuild the cache, your GPU is waiting — and that cannot happen.”

Shetty spoke with theCUBE’s John Furrier at RAISE Summit, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how storage has become the critical path for AI infrastructure, Solidigm’s ecosystem co-design philosophy and the AI Central Lab’s role in producing validated reference architectures at petabyte scale. (* Disclosure below.)

Token per watt drives a new storage design philosophy

The focus on token per watt efficiency is reshaping how Solidigm builds and validates its products. The company’s AI Central Lab — a small-scale replica of a real megawatt data center — runs real AI workloads on NVIDIA H100, B200 and B300 hardware alongside software partners including MinIO, Weka and NetApp, with results that include demonstrated linear performance scaling from one node up to 32 nodes at exabyte capacity, Shetty noted.

“Nobody cares about random read, random write on a data sheet right now,” Shetty said. “What people care about is how does it operate in an AI data center and an AI workload.”

Solidigm’s flagship 122-terabyte D5-P5336 SSD illustrates the density-first argument directly: 24 drives in a single 1U rack yields four petabytes of storage, consolidating what would require 12 racks of hard drives into one — with an 80% to 90% reduction in power consumption. That reclaimed power can be redirected to GPUs, Shadid noted. The company also co-designed the world’s first liquid-cooled SSD with NVIDIA Corp. to support fanless data center architectures, where heat generated by extreme performance is a hard constraint.

“Fundamentally, what lies underneath all of it is efficient storage,” Shetty said. “If you’re looking for density — 122 terabytes, 24 of these in a 1U rack, you get four petabytes of storage. It allows you to have efficiency, as well as opportunity for scale-up as need be. And going into next year, we go double this.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of RAISE Summit:

(* Disclosure: TheCUBE is a paid media partner for the RAISE Summit event. Neither Solidigm, the headline sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE

A message from John Furrier, co-founder of SiliconANGLE:

Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.

15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more

11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network.

About SiliconANGLE Media