Skip to content
AI News HubLIVE
Original source2 min read

Together Compiler: Continual Compilation for AI Inference

Summary

Together AI introduces Together Compiler (TKCC), the first truly continual agent loop for AI inference optimization. It addresses the throughput-fidelity tradeoff with a Tiered Persistent Engine and a cost model, outperforming hand-written Megakernels by up to 1.7x on Llama 1B and achieving up to 2.6x higher TPS than the best open source engines on frontier models, with state-of-the-art batch-size-1 performance of 1111 TPS on M3.1 Flash and 1205 TPS on DeepSeek v4.1 Flash.

SourcearXiv AIAuthor: Reyna Abhyankar, Dylan Lim, Drew Wadsworth, Venkata Hasith Vattikuti, YuTong Julia Zhang, Jung Hwan Heo, Taeksang Kim, Kesavan Ramakrishnan, Evangelene Stanley, Xinyu Fang, Daniel Y. Fu, Simran Arora
Together Compiler: Continual Compilation for AI Inference
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

[Submitted on 10 Nov 2026]

Title:Together Compiler: Continual Compilation for AI Inference

View a PDF of the paper titled Together Compiler: Continual Compilation for AI Inference, by Reyna Abhyankar and 11 other authors

View PDF HTML (experimental)

Abstract:Today's AI workloads are rapidly changing: new models are released more often, model architectures are increasingly diverse, and traffic distributions change frequently. The challenge we face at Together AI is continual optimization for these changing workloads. Compiling a deployment into an optimized forward pass is a challenge that is well-suited for agent hill-climbing. Existing systems struggle with continual compilation due to a tradeoff between measurement throughput and fidelity. Many systems use agents to optimize individual kernels from the inference forward pass using offline benchmarks, but timings are low-fidelity because of data-dependent behavior in primitives like mixture-of-experts (MoE). On the other hand, systems use agents to directly optimize end-to-end engines but suffer from slow engine launch times. To address these challenges, we build Together Compiler (TKCC), the first truly continual agent loop designed for AI inference optimization. To address the throughput-fidelity tradeoff, we propose: (1) a Tiered Persistent Engine that decouples engine launch overheads from forward pass computation; (2) a cost model to select the parallelism configurations that seed the optimization process. We also conduct extensive ablations on agent context management and present our insights on deploying TKCC in production. TKCC outperforms highly-optimized hand-written Megakernels on the Llama 1B workhorse workload by up to 1.7x and the best open source engines on the latest frontier models (Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash) across hardware platforms by up to 2.6x higher TPS on the InferenceX benchmark. TKCC also sets the state-of-the-art performance at batch size 1, with 1111 TPS on M3.1 Flash and 1205 TPS on DeepSeek v4.1 Flash. Finally, TKCC generalizes beyond frontier language model inference. TKCC serves trillions of tokens across our production fleet.

Subjects:

Artificial Intelligence (cs.AI)

Cite as: arXiv:2611.14681 [cs.AI]

(or arXiv:2611.14681v1 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2611.14681

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Reyna Abhyankar [view email] [v1] Tue, 10 Nov 2026 19:37:09 UTC (6,207 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Together Compiler: Continual Compilation for AI Inference, by Reyna Abhyankar and 11 other authors

View PDF

HTML (experimental)

TeX Source

view license

Additional Features

Audio Summary

Current browse context:

cs.AI

new | recent | 2026-11

Change to browse by:

cs

References & Citations

SciX

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • TKCC is the first continual agent loop designed for rapidly changing AI workloads.
  • A Tiered Persistent Engine decouples launch overhead from computation, and a cost model selects parallelism configurations.
  • It outperforms hand-written Megakernels by up to 1.7x on Llama 1B and boosts TPS up to 2.6x on frontier models.
  • TKCC serves trillions of tokens in production and generalizes beyond frontier LLM inference.

Highlights and analysis are generated automatically and may contain errors. Check the original source.