Sticker shock has execs rethinking this whole AI thing

KETTLE Like a drug dealer who's hooked you and raised their prices, business leaders are simply shocked to learn the AI their organizations are becoming dependent on is suddenly a lot more expensive. You can listen to the latest episode of The Kettle right here on this page, as well as on Spotify, Apple Music, or YouTube where you can subscribe to get notified of the latest episode. Kettle host Brandon Vigliarolo is joined by Reg reporter Lindsay Clark and contributor Joab Jackson this week to discuss their recent stories about the rising cost of enterprise AI - and one way a popular open source project is trying to fight tokenmaxxing with tokenminning - but the question remains whether such measure will be enough to prevent cost-benefit analyses from popping that bubble. Will the AI industry adapt to the fact it's still unprofitable, blowback from usage-based billing, and a desire to not pay AI models as much as the human devs they're supplementing or replacing? That's what's on the hob for this week's episode. A lightly edited transcript is below: Brandon: Welcome back to The Register Kettle podcast. I'm Reg Reporter Brandon Vigliarolo, and you're going to be absolutely shocked to discover what we're talking about this week. I'm kidding, of course, it's AI. Specifically the fact that it seems the world is starting to wake up to how much it costs to actually run these giant models that are supposed to make life easier for enterprises and their employees, but it seems like they're leading to some invoice shocks. With me to talk about the latest panic over AI costs are Reg reporter Lindsay Clark and our contributor Joab Jackson. Thanks to both of you for coming on. Lindsay Clark: No problem, thank you. Joab Jackson: Thank you. Brandon: So Lindsay, let's start with a story you wrote recently about the fact that C-suite occupants are apparently having trouble getting a handle on new usage-based AI costs. What exactly has them so confused? Lindsay Clark: I get the impression that big companies are diving in with both feet with AI. It's the latest trend. They're using it for a lot of coding and business apps, trying to do stuff in the business with it. KPMG is a massive global consultancy. They provide IT services and outsourcing services. Brandon: And they're the ones who wrote the report, correct? Lindsay Clark: That's right; they wrote the report and they have some skin in the game. They did a survey of more than 2,000 senior execs over 20 countries and found that 29 percent of them struggled to understand the operating costs as they scale with enterprise AI deployments. Nearly half of them were also looking to re-phase their AI deployments when the costs outweigh the expected value. Brandon: Explain re-phase. Are they rethinking the deployment itself or are they changing the scope? What's that mean exactly? Lindsay Clark: It's just slowing down and looking at what they're doing. They're looking at lower-cost models and high-fidelity models. It's looking for a mix of models to deploy rather than just maxing out on the most expensive ones. Brandon: Usage-based billing seems to be a relatively recent development in this space. It was all free samples until we get you in the door to the point where you're dependent on this, and then we realize we actually have to make money off this. Speaking as an AI frontier lab, we're going to have to charge you per token because we're just not able to turn a profit. Lindsay Clark: Anthropic, OpenAI, and GitHub have all moved from a subscription, flat-fee, all-you-can-eat model to usage-based billing based on tokens. The vendors, both the model providers and the application vendors that want you to use AI agents in their applications, want people to jump in with both feet and use this stuff as much as possible. Then, as is typical for the IT sector, they try to change the commercials as we go. Brandon: It's a big enough issue that more than a quarter of C-suite people are getting bill shock and realizing that they might not be able to afford this. But they're maybe a little bit hooked in because their engineers have been using this long enough. I wrote a story recently about an open source tool that someone wrote to test engineers to make sure they're not losing their edge in this environment, because a lot of them are. We've written plenty of stories about developers becoming dependent on this, forgetting how to do some of the basic things they used to be able to do. It gives these AI companies a big inroad to basically say, "Well, now we're going to actually try to make money." But it does put them in a precarious position. If you charge too much, these enterprise customers are going to try to find a way around it, whether it's an open source Chinese model or some other solution, rethinking their deployments and trying to go with smaller, large-scale models. But if you charge too little, you're never going to make enough money. Is this a needle that these AI labs can thread, or is it one that's pointed straight at the bubble? Lindsay Clark: There's a report from Gartner from a few weeks ago that was quite interesting. They had done some research about the cost on this topic for AI-assisted coding. Brandon: Right, this is one you covered back at the end of June, right? Lindsay Clark: That's right. A researcher called Nitish Tyagi was saying that there's a real lack of transparency from the vendors over the costs of their coding agents and they don't have cost optimization tools that you would expect in the cloud, for example. Because of this, the costs of the coding agent per developer was going to exceed the actual salary of the developer in 2028. That is the average salary globally. He was already finding that in areas of the world where salaries are a lot lower, like India, the cost of agents is actually exceeding the salary of the developer. This is because the cost of agents is the same throughout the world, whereas developers get paid differently according to where they're located. Brandon: I imagine the cost of these are maybe never going to exceed the salary of a developer in Silicon Valley; those guys are making a lot of money. Lindsay Clark: Exactly. Yes. Brandon: When I think about that, there's no way that you could have token costs being in the hundreds of thousands of dollars. But the overall global average is still a lot. I can understand that it's probably why you see a lot of companies concerned about the viability of AI deployments, causing companies to rehire the people they're laying off. It's another thing that screams, "How sustainable is this?" Lindsay Clark: That research, to your point about the bubble, means they have to recoup these costs somewhere. On the macro picture, a big investment house was looking at the capex across the industry for AI datacenters, and it was $1.5 trillion over five years until 2030. That's a lot of money, and it has to come from somewhere. Again, on the Gartner research, they were saying that all the model providers all have kind of different ways of doing the billing as well. There's no standard. So if you're looking at a way to approach this the model providers are likely to, then it's very hard. Gartner were encouraging people to take matters into their own hands, to look at optimizing their own usage, minimizing their own usage. And one of the things that came out of that call was that there was no direct relationship between any increase in token consumption and an increase in productivity and coding in this case. And that was very telling. It's not the case that the more that you use, the more that you get out of it. Actually, they were arguing that if you are careful with how you do this and controlled, then not only do you end up consuming less, but the quality of the code you produce is also higher. That was reflected in a conversation I had a few weeks ago with Spencer Kimball, who's CEO of Cockroach Labs. Spencer Kimball is well known in the coding world, spent a long time at Google, and he wrote the GIMP open source image processing tool. He said at Cockroach they don't do tokenmaxxing; they just use a whole bunch of models, including a lot of open source models and the cheapest models. He said there's no point in maxing out a model when you haven't provided the right context because you'll just get more rubbish back. He was in line with what Gartner had said; he's quite circumspect about how you deploy it on the commercial level. Brandon: Speaking of tokenmaxxing, that brings me to the next story. Joab Jackson, you wrote this a little while ago about a Netflix engineer who wrote an open source tool that has become popular very fast. It's apparently saved many users who have adopted it hundreds of thousands of dollars by trimming input to LLMs in order to save token cost. It's the opposite of tokenmaxxing; this is like tokenminning. What exactly did he come up with? Joab Jackson: This was a home project, as all good open source projects started out as. He had gotten a $287 bill from Claude Sonnet for some debugging and MCP type work he was doing. He was curious how you generate that large of a bill from that modest of a workload. He took a look at what he was sending over to Claude, and it's known that most token consumption is from input. You get charged for both token input and output, but most of the bills come from what you're putting into the system. He was inspecting the stuff that his agent was submitting, and the vast majority of it was completely redundant. It wasn't useful instructions; it was database schemas, JSON templates, or a lot of log work. He figured that if he could create a program that would tear out the redundant parts and then submit the useful information, he could save money. Claude does have a number of settings, but you quickly descend into AWS billing hell. There's a cache setting. Every time you give a query to an LLM, you're tagging on your complete history up until that point, so a lot of the same information gets passed up. You can set that history to stay for five minutes or an hour. This is how things get tricky real quick. You can set it for an hour, but it costs twice as much, though you get ninety percent savings in reads. Now you have to do the math of the stuff you're submitting versus your workload. Brandon: So this is basically trimming the fat, where instead of resending all that stuff as token input, it's basically telling the model, "You've already got this, use what you have." Joab Jackson: It's not even that you're being chatty. It's all these needlessly verbose schemas. If you have it execute a Rust command in verbose mode, you're going to get all that verbose stuff, even though the LLM doesn't need it. So he came up with a bunch of little modules that he calls squashers that look at these areas: database compression, JSON trimming, and so on. There are a lot of VC-backed token trimming tools now, but he wanted something inline that worked directly from the command line. Now, just a clarification, this wasn't work he was doing [for Netflix]; he has another job entirely at Netflix, but he built this program on his own. He let a few Netflix engineers try it and they liked it, but it just took off all of a sudden on its own anyway. Brandon: It's not an internal Netflix product; it's his own project. I'm curious how this would work. I'm assuming that this is entirely dependent on a context window? Joab Jackson: This is shaping the context window, basically. Brandon: If we go back beyond a certain point, because context windows have a termination, I'm assuming that at that point you're going to have to start resending stuff if you're still undertaking the same conversation. This is only effective as long as it remains in context. Joab Jackson: I use Gemini quite a bit, and they do keep a cache of the stuff so it does seem like a conversation. But after a certain point, although the conversation is

[truncated for AI cost control]