AI Tokenomics: How to tokenmin while ROImaxxing

AI Tokenomics: How to tokenmin while ROImaxxing

30.06.26

Enterprise AI

A company accidentally spent $500 million on tokens. We’re not making this up; fact is stranger than your AI agent’s wildest hallucination. And they managed to spend it all in a single month because they hadn’t set employee limits on Claude usage. In a similar vein, Uber and Service Now burned through their entire annual AI budgets in the first few months of this year.

These extraordinary levels of expenditure are driven by dramatic developments along two axes: price and quantity. Major AI providers like Anthropic and GitHub Copilot are moving from generous flat-rate subscriptions to usage-based billing. For example, GitHub Copilot’s June 2026 pricing changes increased model costs for legacy Pro and Pro+ users by as much as 9x to 18x. Also, enterprises are increasingly deploying AI agents at scale, and multi-agent systems typically use about 15x more tokens than chat interactions (meanwhile, agentic coding systems can use >1000x more tokens than chat). And let’s not forget the less obvious pricing/quantity changes, as seen when Anthropic’s Opus 4.7 shipped with a tokenizer which kept per-token rates identical, but generates up to 35% more tokens from the same input.

The good news is that 50-80% of your token spend is unnecessary. There is a multitude of (hidden) ways in which you’re bleeding tokens: everything from using frontier models for trivial tasks, to re-processing the same context repeatedly, to having “gossipy” AI agents who send vast quantities of superfluous information back-and-forth. Reassuringly, all of these problems have fixes, which we discuss in this report.

Here’s our blueprint for token-efficient AI:

Invest in context and memory management solutions. Input tokens, rather than output tokens, account for the larger share of token spend. The best context and memory management solutions feed AI agents the precise information they need (not too little, not too much) in an efficient manner whilst improving accuracy. Also, because these solutions exist independently of model providers, it’s easier to switch models.

Build multi-model systems. Model capabilities are evolving too quickly to bet on a single provider, and Anthropic’s “Fable Fracas” showed that access to frontier intelligence can vanish overnight for geopolitical reasons. In this report, we’ve focused on 3 key components of multi-model systems:

Open source models. As open and closed model performance increasingly converge, ensembles of budget models can rival or exceed frontier intelligence at around half the cost.

AI routers and gateways. For most of the use cases you don’t need frontier intelligence, so routers and gateways direct your traffic to the lowest cost model that can meet your performance requirements.

Inference providers. Agentic AI demands a new inference stack because unlike chat, agentic workloads are long-running, asynchronous, context-heavy, and dominated by tool use. That’s where async and batch inference can tremendously lower costs.

Make verification fast and cheap. About 60% of the cost of agentic software engineering isn’t in initial code generation, but in automated refinement and verification. The “fully loaded” cost of tokens should also include the cost of human verification and re-work.

Focus on workflow design. Even if you run the same agent on the same task, the cost can vary by 30x, and this unpredictability necessitates two actions:

Building real-time visibility into costs and ways to control that expenditure; and

Reserving LLMs for reasoning, not tasks that SQL, rules, or templates can handle. If something can be handled accurately and cheaply using deterministic methods, it’s best to avoid using agents.

Introduce spend controls without stifling adoption, by factoring in actual adoption patterns. Use pooled budgets rather than fixed per-user limits to balance experimentation with cost discipline – e.g. 5% of users may account for 40% of token consumption, and this shifts around over time depending on the pace and depth of adoption.

Don’t optimise for token cost alone. Security, governance and sovereignty are equally important considerations when designing AI systems.

We’ve also featured various startups and scaleups providing solutions (for everything from context and memory management to inference provision) in a market map. If you’re building in this space – or you’re an AI-native company cleverly ‘tokenminning’ while ‘ROImaxxing’ – please reach out to Advika, Simon or Prakriti – we’d love to chat. We’ve been researching on and investing in AI companies for over a decade, and today we have one of the largest portfolios of AI companies in Europe – so we’re clearly very keen on the space.

Note: Certain startups and scaleups may fall into multiple categories and sub-categories; for the sake of simplicity we have assigned each company a single sub-category.

Tech overview

In our previous work Agentic Enablers: Treating AI’s amnesia and other disorders we discussed context and memory management solutions at length.

Context refers to short-term memory that an AI agent has: the prompts you send, the files you attach, the tools you call.

Memory refers to long-term persistence – so you don’t have to constantly remind your AI agent key details about your business or how you want certain tasks to be performed.

Context and memory management improve agent accuracy, which also means you’ll waste fewer tokens and less time fixing any incorrect outputs. Also, an independent context and memory management layer gives you much more flexibility in choice of model provider (this will be particularly important as you leverage cheaper, open source or smaller models – but we’re getting ahead of ourselves).

“Companies want the data mediation layer to be independent from the model provider or agent harness – not only so they can switch between models, but also to enable all their agents to benefit from shared optimisations. It’s inherently better for them to have complete control over their data.” – Matt Henderson, CEO & Co-founder, Coral

Bai et al. (2026) found that input tokens rather than output tokens drive the overall cost of agentic coding tasks, and this is driven by the sheer quantity of tokens consumed. We note that once GPT-5.4 crosses 272K tokens, input prices double and output prices go up for the whole session, so a text-heavy agent can easily drive up the bills. That’s why it’s important to optimise context everywhere – from system prompt to tool calling to file formats to knowledge ingestion, as we detail below:

System prompt: Before the agents even look at the user prompt, the system injects instructions defining their roles, constraints, and output formats. In many frameworks, these system prompts are sent every single time an agent is invoked. Tightening system prompts and moving rarely used guidance into RAG is helpful in reducing input costs by as much as 30%.

External knowledge or web search execution: Poorly implemented web search can flood an agent’s context with full pages, repeated snippets, irrelevant results, ads, navigation text, duplicate sources and bloated citation metadata. The model then wastes tokens reading and reasoning over noise, increasing cost and latency while making answers less reliable. Search APIs solve this by returning cleaner, more structured and more relevant context from fresh, trusted sources. Instead of forcing agents to scrape and ingest messy webpages, they help surface the right information in a compact, deduplicated format. That improves accuracy while keeping token usage closer to the “Goldilocks” zone: enough context to avoid hallucination, not so much that performance degrades through context rot or inflated spend. Startups and scaleups building Search APIs include Exa, Tavily, Parallel, Cala, Linkup, Valyu, Firecrawl, Olostep and Seltz.

Internal knowledge: It’s equally important that internal knowledge is retrieved, curated and fed to the AI models to keep it in the Goldilocks zone we talked about, and here we’re seeing a fascinating variety of approaches emerging. We break these down into different sub-categories:

Retrieval infrastructure: These platforms optimise the retrieval layer itself – how context is fetched and ranked before it reaches the model. Startups in this space include PageIndex (vectorless RAG retrieval) and Ragie (a fully managed RAG-as-a-Service platform for developers).

Context and memory infrastructure: The common thread across these companies is the shift from retrieval-heavy AI to structured, reusable context infrastructure. Rather than asking agents to repeatedly parse raw documents, SaaS data, tickets, transcripts, logs, execution traces, or employee know-how, startups like Hyperspell, Modern Relay, Along AI, Clarifeye, Saphenia, Sentra, Cognee, Mem0, Zep, Xmemory, Prometheux, Stardog, Datalinks turn enterprise knowledge into persistent “company brains” through memory stores, context graphs, ontologies, typed facts, and secure knowledge spaces. Edra turns real-world workflows into reusable playbooks for agents, Signet AI preserves an agent’s identity, memory, and secrets across models, and Mubit learns from past runs so agents can reuse successful approaches instead of reprocessing the same context.

File format: Token costs quickly add up when agents pass around bulky formats like screenshots, verbose JSON, or entire email threads instead of simple text. Screenshots are especially expensive because models have to process every visual detail, while formats like JSON and HTML waste tokens on brackets, tags, and other formatting. The easiest fix is to pass the simplest format that still does the job – plain text, compact tables, or pre-parsed context wherever possible. Tools like iGPT AI help by stripping out duplicate email chains, boilerplate, and other clutter before the model sees it, reducing token costs without losing the information that matters.

Tools gateway: Poorly designed tools waste tokens because agents fail, retry, and carry around far more context than they need. Every extra tool and oversized API response adds to the prompt, increasing both cost and the chance of worse reasoning. The fix is to expose only the tools an agent needs, return concise summaries instead of raw payloads, and cache repetitive data. Over and above providing a tools connectivity and security/governance layer, StackOne’s Tools Gateway does exactly this, reducing huge tool menus to a relevant few, shaping responses for agents, and caching results – cutting token use by up to 90% compared with naive approaches.

Data retrieval: When an agent calls a tool, every data retrieval task involves four steps: (1) finding the right tool, (2) discovering the data, (3) figuring out how to query it, and (4) formatting the response. Most MCP servers, gateways, and coding tools handle the first and last steps well, but leave data discovery and query planning to the agent, forcing it to trawl through APIs and paginated results. Coral shifts that work to the server, letting agents send a single structured query while it handles discovery and cross-source joins behind the scenes, reducing token use by up to 64% on complex tasks.

Context avalanche: That’s what happens when an AI keeps carrying the entire conversation forward, even though most of it is no longer relevant. After 20 rounds of edits, the model isn’t just reading your latest request, it’s also rereading the original prompt, every previous revision, tool output, and abandoned idea. That means five separate requests like “make it shorter” or “add an example” can cost much more than asking for all five changes in one go. The best fix is to batch edits where possible, and occasionally summarise the conversation and start a fresh session so the model isn’t dragging unnecessary context along with it.

Caching and compression: Caching and compressi

[truncated for AI cost control]