How many devs can you fit on a GPU?

This is a first article in a series where we dig into the cost and technicalities of owning your own AI coding agents.

01

Why your token bill blew up this year

The warning signs were everywhere the first half of this year: GitHub Copilot switched to token-based billing [1], Uber burned through its annual AI tools budget in just four months [2][3], and developers reported projected cost jumps from $29 to $750 a month [4]. We even invented words for it: "tokenmaxxing", "token panic", "token budgets".

The median employee [5] currently spends ~$140 per year on AI API usage. Sounds harmless, until you look at the tail: the 90th percentile nears ~$7,300 per year, and the 99th approaches ~$90,000. For many organisations agentic AI adoption is only just beginning, so expect these numbers to change in the year ahead.

Why is the bill so volatile? Because of how we use AI agents: Currently, over 70% of ARR across major model providers [4] comes from coding use cases. The more of your workflow you hand to agents, the higher your token bill becomes.

Many organizations start their AI coding agent adoption using one of the major frontier model providers like OpenAI or Anthropic through API access. A few months in, more and more organizations are at least considering the alternatives for a number of reasons: token pricing models become more costly, users are upgrading to newer and more expensive AI models and agent adoption in the organization starts to boom. There are sensible alternatives out there that are certainly worth a closer look: switching to API routers; using cheaper model tiers for routine work while only escalating the hard 20% to a frontier model and others.

However, if your token consumption becomes significant or if you're working with sensitive data, you might want to abandon the API approach altogether and setup your own inference stack. When ownership of hardware is on the table, you'll need to understand what renting or owning a GPU actually gives you in return. When does it make sense to make that switch? Read on as we give you the tools to decide for yourself.

02

You pay for the pool 24/7, your team uses it 8 hours a day

So you've decided to investigate whether buying or leasing your own GPUs makes sense. Once configured correctly, it will give you an API endpoint you can plug into your coding tools, much like Anthropic, OpenAI or other model providers offer.

In contrast to the frontier APIs, your own endpoint has a ceiling on output tokens per second but you only reach it when the system is fully loaded. One developer working alone leaves most of that ceiling unused, but the same hardware serving forty-eight agent sessions in parallel is a different story. Your ceiling is determined by the model, the GPU(s), and the serving configuration.

Fig. 01

Total tokens/sec vs concurrent users

Illustrative example of a model A running on a hardware setup B.

Unlike an API usage model where you are charged per token consumed, running your own setup signifies you pay for the underlying infrastructure and the cost of keeping it running. In all but a few cases that means you're also paying for it even when you're not generating tokens. Whether that trade-off works for you, is decided by two questions. The first is how much of the pool are you using. Let’s tackle that one first.

You size for the peak and pay for it 24/7. Utilisation, not headcount, is what makes or breaks the cost case.

Real demand is not constant, but spiky. For many organizations, tokens for coding tasks will be near-zero overnight, ramp up through the morning, dip visibly at lunch, and peak again mid-afternoon. This means you buy hardware for the peak load, but you pay for it 24/7 (unless you rent ‘GPU spot instances’, but these are harder to rely on in this case). The shape of your peak in usage matters as much as the volume: a single-timezone team concentrates token demand into narrower spikes than the same headcount spread across timezones: same tokens, higher peak usage, but spread out longer. Published data on enterprise inference workloads serving internal developer tools reports average GPU utilization of 15–22% [6][7], though even a well-run deployment currently rarely exceeds 25–35% (and that is being generous). The graph below shows the real, actual token usage over 2 days a couple of weeks ago from our friends over at TechWolf[14], reflecting exactly what we describe above.

Fig. 02

Real-life token usage is spiky and different to each organisation

One emerging trend may work in your favor here: As organizations mature in their agent driven workflows, more and more sessions are starting to come from automated agents rather than people (scheduled jobs, agents autonomously picking up a bug report at 3am, etc.). This allows for work that doesn't adhere to office hours. If scheduled with intent, these automated sessions can soak up overnight capacity you are paying for anyway.

Now let’s tackle the second question, how many developers can share the pool before it feels slow?

03

What happens when 48 developers share a box?

A token pool occupied by just one developer feels wonderful - tokens stream in faster than you can read. The question is what happens at five, ten or twenty concurrent developer sessions, because AI coding agents are greedy, bursty clients that not only consume tokens, but also browse the web, execute files or compile code.

One of your main concerns when moving away from a frontier model provider API should be developer experience. When looking at benchmark results you will often see tokens-per-second used to gauge how well a model performs on a given GPU setup. While it is an easy to understand metric, we find it less useful in this context. After all, you can burn thousands of tokens with no satisfiable result, or you can switch to a smaller model that ends up using 10 times more tokens.

The better unit of value delivery is a task successfully completed, and while that is harder to quantify in general, that's what we'll often return to in the analysis below. You will want to understand how long a task takes start to finish and how fast tokens stream while the agent is waiting on the model.

The better unit of value delivery is a task successfully completed.

To make task completion measurable we use a subset of tasks from SWEBench Pro by ScaleLabs, a popular set of longer-horizon code engineering tasks. While it is most often used to pit individual models against each other, our team is using it specifically to create a more lifelike and challenging enough workload as we compare the cost and performance of different model and hardware combinations under real use cases that would make sense for your organization.

A couple of graphs will help us further evaluate the performance of the model & hardware combinations we have chosen. First, we run our subset of coding tasks using a given model on a specific hardware setup at increasing levels of user concurrency and evaluate how this impacts the time to complete individual tasks. This will help to find that sweet spot of how many concurrent users you can have active for your chosen setup before tasks simply start taking too long to complete.

Fig. 03

Minutes per task vs concurrent users

Illustrative example of a model A performing coding tasks on a hardware setup B.

It is convenient to see this information together. So often the trade-off between how much the hardware is in use vs. the developer experience in terms of speed is shown in one graph that allows for convenient decision making. We’ll use the graph below to show how different model / hardware combinations approach the maximum token output they can generate as user concurrency increases, while keeping an eye on how long we can keep our developers wait for a task to be resolved.

Fig. 04

Tokens/sec vs median task time

Illustrative example of a model A performing coding tasks on a hardware setup B. C means number of concurrent users.

Understanding these values will set you up to better understand the differences in performance and developer experience across the options that may be a match for your organization’s needs.

Over the last weeks our team was hard at work to do ~100 runs of a selection of 64 real coding tasks from SWEBench Pro across different and appropriate hardware configurations to calculate precisely the right metrics (median time per tasks, tok/s/u and many more). Guided by open-weight quality rankings from Artificial Analysis [9], we paired each hardware tier with a leading model that fits its memory budget:

Qwen3.6-35B-A3B-FP8 on a single H200 served with vLLM

DeepSeek-V4-Flash on 4×H200 served with vLLM

GLM-5.2 on the 8×B200 rack served with SGLang.

Qwen3.6-35B-A3B-FP8 on NVIDIA's DGX Spark served with vLLM

In the next sections we’ll dive into the actual results and round it up with a cost analysis comparing buying your own GPUs to renting them to buying tokens from either the frontier model providers as well as open weights API providers. We feel these are the kinds of questions many organizations are pondering at the moment and we wanted to give you the tools to answer them.

04

From a Spark on a desk to an 8×B200 node: which box to buy

If you’re interested in renting or buying hardware, several options are available. Below we introduce four popular contenders that we’ll come back to throughout this post. Each of those also comes with a best-in-class open weights model that nicely fits the hardware.

You’ll see that we’re leaning heavily into NVIDIA here, which has a lot to do with what was easily available to us. There are definitely similar alternatives from other suppliers, which we’ll get to in later posts. For now, consider these as reference choices rather than specific endorsements.

Let’s start with the cheapest option, an NVIDIA DGX Spark. This is the box for a team that wants to try self-hosting without talking to procurement: it sits on a desk and costs less than a conference trip. It can fit Qwen3.6-35B-A3B (referred to as "Qwen3.6"). That’s a surprisingly big model for such a machine - We’ll see later in the article what it delivers in terms of developer experience.

Next up is the NVIDIA H200, a powerful, single accelerator. Perhaps surprisingly, It actually does not buy you a lot more in terms of the models you can host compared to a DGX Spark. This is mostly determined by the available memory (128GB for the DGX Spark vs. 141GB for the H200). However, what you do get is a bigger token pool which translates in throughput speed for higher concurrencies.

If you want to host more powerful models, but don’t want to fully break the bank, NVIDIA HGX H200 with 4xH200s is a solid option. With 4xH200s, you can host bigger models such as DeepSeek-V4-Flash, which promise a very good compromise in terms of speed, concurrency and quality.

Taking it up a notch, the HGX B200 with 8xB200’s is a powerful reference system, as it lets you serve state-of-the-art open source models like GLM-5.2 that currently come closest to the frontier models in terms of quality. This is your option if quality is the main concern.

At the time of writing this article, there is a lot of buzz about Kimi’s K3 model, as early benchmarks (which will need to be confirmed) indicate it would be able to compete with Anthropic’s Fable model, but at much lower cost. For now, our team is awaiting the open weights release for this model later this month to evaluate what it would take for this 2.8T to run on your own infrastructure. More on this soon.

Next up, we’ll put these contenders to the test in terms of developer experience, cost, and quality of the models they can fit. Let’s do this.

05

How big are those pools?

So what about having a DGX Spark sitting on your desk within reach? Well, unsurprisingly, it does not come with a big token pool. In our tests, we could not push it beyond 1-2 concurrent users and were con

[truncated for AI cost control]