The new economics of AI cost management | Databricks Blog
Skip to main content
Enterprises are shifting from "tokenmaxxing" to "valuemaxxing," measuring the impact generated per dollar spent on AI.
Measuring AI ROI requires attribution granular enough to trace spend to a user, team, project or even a single pull request
Spend guardrails, end-to-end observability and a centralized point all AI traffic flows through are the foundation teams need now
For the past two years, the story around enterprise AI adoption has mostly been about getting people to use it. Over the past few months, that story has rapidly been changing in many organizations. AI observability that once measured success by token volume is starting to measure success by return on investment, and the enterprises that figure out cost management first are the ones that will be able to scale AI without their spend outrunning their revenue.
At its core, AI cost management is the practice of measuring, attributing, controlling and optimizing AI spend across models, agents, applications and teams. The goal isn't simply to use fewer tokens, but to maximize the business value generated for every dollar spent. As AI agents take on more autonomous work, that discipline increasingly extends to managing the cost of agent reasoning, tool calls, and the models powering them.
We sat down with Kevin Stumpf, a Senior Staff Engineer at Databricks and Technical Lead for Unity Gateway, to talk through how teams should actually put that discipline into practice: not treating AI cost as a line item to minimize, but as a system to manage.
From tokenmaxxing to valuemaxxing
Social media is full of leaders arguing that "tokenmaxxing," running AI as hard and as often as possible, is the right way to accelerate innovation. Other than the obvious cost implications, why is that the wrong way to think about AI adoption?
Kevin Stumpf: The industry is moving from tokenmaxxing to valuemaxxing. In the early days, there were public leaderboards showing how many tokens people were burning through, and that made sense as a strategy at the time. A lot of people were reluctant to start using AI, so showing that the leaders everyone looked up to were heavy AI users gave others a reason to follow. If I see someone I respect using a ton of AI, I want to be like that person.
But the system got gamed. Once being at the top of the leaderboard became a status symbol, people figured out the fastest way to get there wasn't good work, it was volume. Spin up a dumb agent that runs in a loop, have it ask what the weather is over and over, and burn through tokens doing essentially nothing. That was the predictable outcome of an incentive structure that rewarded raw usage instead of outcomes.
Now that organizations are broadly bought into AI, the conversation has shifted to responsible, effective use. What we should really be building toward is a different kind of leaderboard, one that ranks people by the impact they generated with AI divided by what it cost to generate it. That's return on investment, and it's a much harder thing to quantify than raw token counts, but it's the number that actually matters. Related: What’s Driving Enterprise AI Adoption?
What cost attribution really means in practice
Can you walk us through what "cost attribution" really means in AI?
Kevin Stumpf: AI cost attribution is ultimately about answering two questions: where does my spend come from, and what did I get for it? At the simplest level, that means attributing cost to an individual user or a team, like a data science team or a software engineering team. It can also map to a specific project, or to a node in an org chart, like understanding how much spend sits under a VP of AI versus an SVP of engineering.
The goal is granular enough visibility that you can slice and dice the data and understand where cost is actually coming from, because you can't manage what you can't measure. Databricks provides cost attribution out of the box by user, group, agent, AI gateway endpoint, model, provider and time period. But organizations can also hyper-customize it using request-level tags. A tag can represent anything your organization cares about, like a specific pull request, a feature or an external customer, and once every request carries that tag, you can aggregate spend against it however you need to.
That shift in customer sentiment has been real. Six months ago, the top concern was falling behind on AI adoption. Now it's the opposite: AI is often used for silly things, costs are growing faster than revenue, and leaders are asking how to bend that cost curve before it eats into profitability. This isn't a hypothetical. It's a boardroom-level conversation at nearly every organization we talk to, including internally at Databricks.
Taming context bloat
Token usage and context window size get blamed a lot, but are rarely explained well. What does context bloat actually look like in a production system, and what's the practical fix?
Kevin Stumpf: Context bloat means throwing far more information at an agent than it needs to make a good decision. That's a problem for two reasons. First, it burns unnecessary tokens. Second, it's distracting: the agent has to process that extra information and decide whether it's relevant, which is its own kind of overhead. It's not unlike handing a person a stack of irrelevant documents and asking them to figure out what matters before they can even start the task.
The best fix isn't trimming or caching after the fact, it's not generating the bloat in the first place. A common and effective pattern is letting the agent pull relevant context on demand, through tools, MCP servers or skills, rather than proactively pushing everything it might need up front. You should only push context to an agent by default if you're 90% certain or more that it's genuinely important for the task at hand. Otherwise, let the agent know what's available and allow it to fetch what it needs when it needs it.
Dynamic model and harness routing
Model selection is often a "pick once" decision. Why should it be dynamic instead, and how does intelligent routing decide, in real time, whether a task needs a frontier model or something cheaper and faster? Does that same idea extend to choosing between different agent harnesses?
Kevin Stumpf: AI cost optimization isn't about always choosing the cheapest model, and teams should shift to choosing the lowest-cost model or harness that can deliver the quality a task requires. The reality right now is that frontier models are highly capable and expensive, while non-frontier models are still very capable, just less expensive. Roughly 5 to 10% of tasks genuinely need a frontier model, either because the stakes are high enough that you want the best possible outcome, or because the problem is too complex for anything less. The other 90% of tasks are solvable by lesser models, whether that's open source models you host yourself or cheaper model families from the major labs.
Left to their own incentives, most employees will default to the most expensive model available, because their goal is simply to get the problem solved, and nobody is watching what it costs them individually. Smart Routing solves that by inserting a layer that assesses task complexity in real time and decides whether a request should go to a frontier model or a lower-cost one, with the goal of maintaining the same quality output at a lower cost.
Harness routing is the next layer beyond model routing. There's emerging research showing that the exact same model, run through a different harness, can produce meaningfully different costs. That makes sense, because the harness determines how the model interacts with its environment: what rules it follows, when it compacts the context window, what business logic governs its behavior. All of that directly affects how many tokens get spent. Harness routing is the next step in that crawl-walk-run evolution of cost optimization, which we enable through Unity Gateway and Omnigent, our meta-harness.
Designing guardrails on spend
Let's talk about guardrails on spend itself, like hard caps, budgets, and rate limits. What's the right way to design these so they stop runaway costs without hurting developer productivity or breaking production workflows?
Kevin Stumpf: Some rules apply to every organization, and others depend on the use case or the company. Everyone should have hard spend caps in place to catch runaway usage. If your typical power user spends around $300 a day on AI at most, set the hard cap somewhat above that, maybe $500. That gap gives you room to catch genuine errors, like an agent that got left running in a loop by accident, without capping off legitimate high-usage days.
Beyond that baseline, differentiated budgets by team or project make a lot of sense. A team working on something critical for the CEO might warrant a much higher budget than a team working on lower-priority work, and that decision should weigh the value of what that team produces against the cost of giving them more headroom.
Budgets should also be able to trigger different types of responses, not just a hard stop. They can send alerts, apply hard caps or reduce available model capacity, for example cutting off default access to frontier models once a team has burned through 70% of its budget and routing the rest of their requests to open source models instead. The core principles that apply everywhere: prevent runaway spend, set differentiated limits by group or project, alert before you cap and build in intelligent capacity reduction rather than an abrupt shutoff.
Read now
Measuring what AI actually delivers
Which platforms offer agent monitoring and cost tracking together, and how far can that visibility take you toward measuring actual ROI, not just spend?
Kevin Stumpf: Measuring cost is a lot easier than connecting that cost to return on investment, and that's the question everyone wants answered and almost nobody has fully solved yet. Session or request-level tagging is what makes this possible. If an agent is working on a specific pull request, it can tag its session or specific requests it generates with that pull request's identifier. That attribution flows back into a cost dashboard, so you can see, for example, that a given pull request cost $300 to complete. This is something we’re already doing internally at Databricks.
In an ideal state, and this is something we're building into the Unity Gateway product. You can go a step further and connect that pull request to a specific feature, and that feature to a measurable outcome, like how much revenue or pipeline it generated in a given period. That's the holy grail: not just knowing what AI cost, but knowing what it returned. With Unity Gateway, we already have a highly customizable platform to build this on, and we're investing in making that cost-to-return view available out of the box rather than something teams have to construct themselves.
Spend visibility requires a centralized system
For a company running multiple models, agents and tools across many teams, what does good spend visibility actually require? Is it a dashboard problem, a governance problem or a FinOps problem?
Kevin Stumpf: Good spend visibility requires that every AI interaction flows through a centralized system. That's what allows you to lock down the right information consistently, so you can build reliable dashboards and alerts on top of it. An AI gateway provides a centralized control point for AI cost management across models, agents, tools and teams. Databricks Unity Gateway is designed to provide that centralized control point across models, agents, tools and coding agents. When AI interactions flow through a common control point, organizations can observe usage, attribute costs, enforce budgets and optimize spend in one place.
But having the centralized system isn't enough on its own. You also need a way to enforce that everyone actually routes their AI traffic through it. Visibility only works if adoption is universal, not optional.
Balancing cost optimization and output quality
Cost control can't come at the expense of quality or reliability. How do you know when an optimization has gone too far and started hurting output?
Kevin Stumpf: That's genuinely use case specific. You need a feedback mechanism, and it can be either subjective or objective. Subjective feedback is anecdotal, someone on the team telling you they can't get the job done within the current budget. That's fast to get but easy to dismiss as one person's complaint. Objective feedback comes from automated evaluations, which are harder to set up but far more reliable.
For a common use case like coding agents, that might mean defining a proxy metric for quality, such as merged pull requests that don't get rolled back within two weeks. From there, you can experiment: cut the budget in half and see whether that gold star metric moves, holds steady or even improves, since constraints sometimes breed creativity. Running parallel experiments across teams with different budgets, models and harnesses, or time-bound comparisons over a few weeks, lets you infer whether a given budget is creating value or destroying it, rather than guessing.
The value of end-to-end tracing
What role does end-to-end tracing and monitoring play here? Once you can see prompts, tool calls and token usage across an entire agent workflow, what does that unlock that you couldn't do before?
Kevin Stumpf: Tracing is about collecting the full exhaust of what agents are doing: what they're thinking, what tools they're calling, what data they're pulling in, whether the prompts are good or bad. That raw data is what you can mine for insight, whether that's identifying abuse, waste, security leaks or simply patterns you never thought to look for.
This mirrors what happened in the broader data industry. Early on, we only had clean, structured data in databases. Then came data lakes, which for a while became data swamps because people dumped unstructured data into them without a way to extract value from it. Eventually those swamps turned back into usable data lakes, and now we're in a better position than ever to extract insight from unstructured and semi-structured data, largely because LLMs are flexible enough to process it. Agent traces fall into that same category. The more of it you collect, the more insight becomes available, and it's easier to get to than it's ever been.
Scaling to agent fleets
A lot of enterprises are moving from single AI applications to fleets of agents. Does that shift change the cost equation (new cost drivers? new blind spots?) compared to running one model behind one app?
Kevin Stumpf: It doesn't change the equation so much as it multiplies it. Agent swarms are going to be the new reality, and every one of those agents will be spending tokens that need to be attributed, monitored and managed. Right now is the time to put the right cost visibility and management foundation in place, before the number of agents in production grows the way it's projected to.
Coding agents are the dominant category burning tokens today, but that's going to expand into many other classes of agents doing autonomous work around the clock, agents calling other agents calling other agents. Organizations that wait to build centralized AI observability and cost management infrastructure until that complexity arrives will be building it under pressure, instead of on their own terms.
If you had to leave enterprise AI leaders with one mindset shift about cost management heading into next year, what would it be?
Kevin Stumpf: I wouldn't wait for next year. This is urgent, and it should be treated that way starting now. If I had to boil it down to a single idea, it's this: you can't manage what you don't measure. That applies to raw materials, it applies to people, and it applies to AI just as much. AI is quickly becoming one of the most important line items on a company's cash flow statement, and it needs to be fully instrumented so it can be measured and, in turn, managed. It's just another raw ingredient in the system that produces your organization's goods or services, and it deserves to be treated with the same discipline.
The clock starts now Controlling AI costs across teams using multiple models and tools isn't a problem that gets solved with a single dashboard or a one-time model choice. It's an operating discipline: attribution granular enough to trace spend to a person or a project, routing that adapts in real time to what a task requires, guardrails that catch runaway spend before it becomes a headline and tracing detailed enough to turn raw usage data into real insight. The organizations getting ahead of this aren't the ones spending the least on AI. They're the ones who can say, with confidence, exactly what they got for what they spent.
Learn more about how Unity Gateway centralizes cost attribution, smart routing, and spend guardrails across your AI estate.
Get the latest posts in your inbox
Subscribe to our blog and get the latest posts delivered to your inbox.
Sign up
View all blogs