Skip to content
AI News HubLIVE
More
Summary only

Cohere's North Mini Code Megakernel Serving Engine

A new serving engine from Cohere wraps the entire decode step for the North Mini Code model in a megakernel, delivering 1.25–1.41x faster end-to-end throughput than vLLM on an H100 and 62% of memory-bandwidth speed of light at batch size 1, without accuracy loss.

SourceCohere Blog
Cohere's North Mini Code Megakernel Serving Engine
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions

Summary only

Under the source content permission, this page provides an editorial summary, key points, and a link to the original article.

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • Megakernel runs the whole decode forward pass as a single persistent kernel, avoiding launch/sync stalls that leave HBM bandwidth unused.
  • Outperforms vLLM by 1.58x at batch size 1 (292 vs 185 tok/s) and sustains the gain across batch sizes and up to 256K context.
  • Production serving features: continuous batching, paged attention, ragged sequence lengths, OpenAI-compatible API and tool calling.
  • Single-CUDA-file implementation on GitHub with a porting recipe; no compiler or new programming model needed.

Highlights and analysis are generated automatically and may contain errors. Check the original source.

More recent stories on this topic · Models

Cohere's North Mini Code Megakernel Serving Engine | AI News Hub