Skip to content
AI News HubLIVE
More
Summary only

Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Agents on SGLang

Tsinghua and CUHK researchers have open-sourced AKV, an SGLang-based serving system that lets LLM agents drop and reposition KV cache entries instead of summarizing history, cutting average cost by 27.7% across four agent benchmarks.

SourceMarkTechPostAuthor: Asif Razzaq
Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Agents on SGLang
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions

Summary only

Under the source content permission, this page provides an editorial summary, key points, and a link to the original article.

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • AKV extends SGLang's radix tree with drop and reposition nodes, reusing retained KV states without re-prefill.
  • Across three models and four agent benchmarks, average cost fell 27.7% and average score rose 1.7 points versus threshold summarization.
  • At concurrency 16, P95 request latency dropped from 229.90 seconds with the Standard baseline to 30.63 seconds with AKV.
  • The Apache-2.0 release currently supports only Qwen, GPT-OSS, and MiniMax-M2 text models.

Highlights and analysis are generated automatically and may contain errors. Check the original source.

More recent stories on this topic · Models