Fast token generation emerges as the key differentiator as heterogeneous inference takes hold

The race for fast token generation has moved from benchmark sheets into production data centers, and the hardware blueprint for winning it is no longer a GPU-only story.

As agentic AI use cases multiply and users demand real-time interactivity, inference infrastructure is being redesigned from the rack up. The divide between compute-heavy prefill and latency-sensitive decode is forcing a new class of purpose-built accelerators into the picture, according to Sid Sheth (pictured, right), co-founder, president and chief executive officer of d-Matrix Corp.

“It is the first public announcement of a heterogeneous compute solution targeted specifically for fast token generation in partnership with NVIDIA,” Sheth said. “We have a lot of inference clouds coming to us and saying — more low latency, deploying that in a GPUs-only infrastructure just doesn’t get us there.”

Sheth and Sudeep Bhoja (left), co-founder and chief technology officer of d-Matrix Corp., spoke with theCUBE’s John Furrier at RAISE Summit, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed d-Matrix’s Corsair production launch, the Parasail-NVIDIA partnership, and the company’s next-generation 3D memory architecture. (* Disclosure below.)

Fast token generation and the heterogeneous inference architecture

The announcement — a Parasail deployment pairing d-Matrix Corsair accelerators with NVIDIA Hopper and Blackwell GPUs — marks one of the first commercial-scale examples of heterogeneous disaggregated inference in production. The economics behind it are straightforward: premium fast tokens are currently priced as much as 10x higher than standard throughput tokens, creating a new revenue tier that inference providers are racing to capture, Sheth noted.

“What we’re really unleashing here is a new class of fast tokens,” he said. “Fast tokens today equals premium tokens. You have Anthropic Claude Code today — they have something called Fast Mode, and in that mode you essentially have much higher levels of interactivity with the application. The application developers are charging more for those fast tokens.”

The technical case for heterogeneity rests on memory bandwidth. AI inference is bottlenecked not by compute flops but by how fast data can move between memory and logic, Bhoja noted. By stacking DRAM and logic together on a single substrate, d-Matrix’s Corsair platform delivers memory bandwidth well beyond what high-bandwidth memory architectures can achieve.

“It’s all about memory bandwidth — having enough memory capacity and then getting the bandwidth out of the memory,” Bhoja said. “If you can combine this into a single substrate, either in three dimensions or on a single piece of chip, you can make memory bandwidth much faster because the distances that we’re moving are much closer. What we offer at d-Matrix is to stack DRAM and logic together in a single chip, and doing that we get many times the performance of HBM memory — and because we don’t move many millimeters on silicon, we can do that at very low energy.”

Looking ahead, d-Matrix is developing a next-generation 3D architecture that takes the memory-compute integration further — hybrid bonding four DRAM stacks directly on top of compute to multiply capacity and bandwidth in a smaller footprint, Bhoja said.

“We want to put four DRAM stacks on top of compute and get the capacity up, the bandwidths up,” Bhoja said. “You get essentially many more fast tokens in a smaller footprint.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of RAISE Summit:

(* Disclosure: TheCUBE is a paid media partner for the RAISE Summit event. Neither Solidigm, the headline sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE

A message from John Furrier, co-founder of SiliconANGLE:

Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.

15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more

11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network.

About SiliconANGLE Media