Skip to content
AI News HubLIVE
Summary only

LightSeek Foundation Releases TokenSpeed, an Open-Source LLM Inference Engine Targeting TensorRT-LLM-Level Performance for Agentic Workloads

TokenSpeed is an MIT-licensed open-source LLM inference engine designed for agentic workloads. Its architecture features a compiler-backed parallelism model, a C++ FSM scheduler for safe KV cache management, and high-performance MLA kernels. On NVIDIA B200, it outperforms TensorRT-LLM by ~9% in min-latency and ~11% in throughput. The engine is currently in preview.

SourceMarkTechPostAuthor: Asif Razzaq
LightSeek Foundation Releases TokenSpeed, an Open-Source LLM Inference Engine Targeting TensorRT-LLM-Level Performance for Agentic Workloads
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions

Summary only

Under the source content permission, this page provides an editorial summary, key points, and a link to the original article.

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • TokenSpeed is open-source (MIT license) and tailored for coding agent workloads with long contexts.
  • Its scheduler uses compile-time safety for KV cache management, separating control and execution planes.
  • Benchmarks on NVIDIA B200 show TokenSpeed surpassing TensorRT-LLM in both latency and throughput.
  • TokenSpeed's MLA kernel has been adopted by vLLM and nearly halves decode latency.

Highlights and analysis are generated automatically and may contain errors. Check the original source.

More recent stories on this topic · Models