A Data-Driven Explanation: Why Do AI Agents Still Fail

Jeremy Tian Aug 04, 2026 It’s been about two years since AI agents went mainstream. According to Capgemini’s 2025 report, AI Agents were estimated to generate roughly $450 billion in economic value across 14 surveyed countries by 2028. That’s less than a year and a half from now. So, how are AI agents doing now? Coding agents: remarkably well. Claude Code, Codex, and others have crossed the threshold from autocomplete to shipping real work end-to-end. Anecdotally, I know many people who don’t even open their IDEs anymore. But it turns out, coding agents are the exception. According to Stanford’s 2026 AI Index, organizational AI adoption continues to rise, but AI agent deployment remains in the single digits across almost all business functions. For AI agents specifically, researchers from Stanford, Berkeley, IBM, etc. found that 68% of deployed agents execute no more than 10 steps before human intervention. It’s tempting to mentally extrapolate improvements across AI: “if AI can handle something as hard as disproving the Jacobian conjecture (an 87-year-old math conjecture), surely it can handle my back-office workflow.” This line of reasoning is in fact so common that there is a term for it: the “Jagged Frontier” of AI, which describes the imbalance of AI capabilities. An interesting example to illustrate this is how Gemini Deep Think performed well enough to earn a gold medal at IMO, but can only read an analog clock correctly 50.1% of the time. None of that explains why enterprise agents underdeliver, though. The short answer is that making AI agents that consistently perform well is very hard. In general, four things explain most of the gap: variance, benchmark quality (datasets & evaluations), agent-specific errors, and alignment. What follows is a data-driven deep dive into each and why there’s such a massive gap between the incredible enterprise AI Agent value we were promised and the little value that has been provided. On Variance & Reliability Your Agent’s Performance Ceiling and Floor Solving reliability in a non-deterministic system is inherently a really hard problem. To illustrate this, let’s model an AI agent as a system that performs some number of steps autonomously to complete a task. Let’s assume the probability that any step succeeds is 90%. When your agent attempts a workflow that takes 10 steps, only 34% of your agent runs will succeed on all 10 steps (0.9¹⁰). When that process doubles to a 20-step workflow, the end-to-end pass rate decreases to 12%. A 30-step workflow has a 4.2% success rate. End-to-end reliability decays as the number of steps grows. Because every step is now non-deterministic, the overall reliability of the system decreases exponentially with the number of steps in your workflow. This is a big problem that gets exacerbated by complexity. Even if we increase the probability of a single step succeeding to 99%, after 30 steps, you can only expect 73.97% of your end-to-end runs to complete successfully, which is a massive downgrade from the 99% individual step success rate. Of course, real agents aren’t this simple. The calculations above assume that each step is independent of each other. Subsequent steps are not independent. In reality, errors compound (a bad step corrupts context for every step after it). But on the flip side, agents can also recover from their mistakes, you can build guardrails to guarantee specific output formats, etc. This example is just a first-order approximation, modeling an AI agent extremely simply. Unfortunately, what is not just a first-order approximation is the idea this example illustrates: more steps means less reliability, fast. The problem we just described was “intrarun variance”; in other words, the variance within a run. But your agent doesn’t just run once. It will need to solve the same or similar tasks many, many times. Intrarun variance’s counterpart is interrun variance—the variance across multiple runs. Sierra knew this was a problem, which is why when they built τ-bench (one of the industry standard benchmarks for customer support), they made pass^k (pass-to-the-k) a first-party measurement. Pass^k, simply put, is just a measurement for interrun variance: run each dataset task k times; a task only counts as passed if it succeeds all k times. In other words, pass^k asks what your agent can do every single time, not just once. To solidify the issue of interrun variance in a more concrete example, let’s take a look at τ³ bench (an improved τ-bench). For τ-knowledge (banking domain), the highest performing model was gpt-5.2 (high reasoning) with a pass^1 of 25.52% (pass^1 here means they ran each task in the domain once, and 25.52% of them passed). At pass^4, it dropped by about half to 13.40% (pass^4 here means they ran each task in the dataset 4 times and only 13.40% of tasks passed all 4 times). And note that this is not an old-model problem: gpt-5.2 is a frontier reasoning model, and interrun variance still cuts its performance in half. Pass^k tells you how consistent your agent is. A natural follow-up question is: how capable is it? You can measure this with pass@k: of k runs, if at least 1 of the k runs passed, it counts as a success. You can imagine pass^k as an intersection of the successful runs and pass@k as a union. Looking at the graph below, all the models follow the same pattern, so let’s just focus on gpt-4o (orange). The dotted line shows the accuracy for pass@k (max current capability) across k trials. As you can see, the dotted line starts at ~60% but quickly approaches ~95%. On the other hand, the solid line shows the accuracy for pass^k (consistency measurement) across k trials. Much like the banking example above, the solid line quickly decays to around half of the original accuracy. Why does this matter though? Think of pass@k as the ceiling. What the model can do on its best attempt. pass^k is then the floor; i.e., what you can actually count on in production. For gpt-4o on τ-bench, the ceiling is ~95%, and the floor is ~30%. This matters because a customer whose refund gets botched doesn’t care that the agent would have gotten it right the other 70% of the time. At this point, you might be thinking: if the ceiling is 95%, why not just run the agent k times and take the run that succeeded? In some domains, this actually works. Coding is (again) the lucky one: you can run the tests and know which attempt was correct. But most agentic tasks don’t come with an answer key. In production, nothing tells you which of the k runs was the right one. If you could automatically detect successful runs, you would have already solved agent evaluation (more on that later). And worse, agents act on the world, which isn’t always so forgiving. You can’t process a refund 4 times and keep the best one. So, when you see that gpt-4o has an accuracy of ~60% on τ-bench, maybe the first instinct is that we need to make the agent better. But the pass@k shows that the 40% performance gap is almost all explained by variance, not agent/model capability. In other words, gpt-4o can already solve 95% of the problems. Just not consistently. This pattern can be seen across almost all public benchmarks as well as with AI agents of customers we have worked with. Variance is one of the massive bottlenecks of AI agents—that is both intrarun and interrun variance. And it has one more consequence we haven’t talked about yet: if a single run is this noisy, then a single benchmark score is a noisy sample too. Variance doesn’t just make your agent unreliable; it makes your measurements of your agent unreliable. This brings us to the next big problem: datasets and benchmarks. On Benchmarks Your Dataset Might Be Lying to You Variance makes your scores noisy. But there’s a second, sneakier problem: the dataset you’re training and evaluating your agent on can itself be wrong. In 2016, a 19-year-old MIT student and a 21-year-old Carnegie Mellon dropout hopped on a flight to San Francisco—a flight that would change their lives forever. They had just gotten accepted into Y Combinator, and their company converged on a simple idea: machine learning was not only driven by compute, but also by better training data. They found early success with autonomous-vehicle companies, and eventually grew into other industries like robotics, defence, government, and more. Nine years later, Alexandr Wang and Lucy Guo’s company, Scale AI, was valued at ~$29 billion—proof that in the age of AI, data is just as valuable as compute. We believe the same about data for AI agent development. A common misconception is that if you’re not fine-tuning a model, you don’t have training data. But your agent is still trained on a dataset (just not through gradient descent). Every time you adjust a prompt, rewrite a tool description, or restructure a workflow because of how your agent performed on your dataset, you are training your agent on that data. The dataset is the signal your entire iteration loop is steered by: it decides which experiments look promising, which regressions get caught, and which “improvements” ship. Needless to say, the data you train on is quite important; in fact, some would argue that your agent will only be as good as the data that it’s trained on. Even if your agent performs well on your dataset, it will perform poorly in production if the dataset is stale/outdated, or if it’s not from the same distribution. Because of this, both the quality of and the distribution of dataset items matter a lot. Public benchmarks are broken more often than you’d think Unfortunately, datasets are hard to make well. Even Sierra’s industry-standard τ-bench dataset was revamped into a τ³ dataset. Between the original and the improved dataset, 53 tasks out of 164 total tasks needed to be updated for reasons like “Impossible or Contradictory Constraints”, “Incorrect Expected Actions”, etc. That’s over 30% of the tasks of the original benchmark. WebArena had a similar problem: its 812 original tasks needed a substantial round of repairs (e.g. misaligned evaluators, clarified ambiguous instructions, replaced brittle matching rules). Following this, false negatives from these issues were reduced by 11% in their baseline agent. The same dataset quality problem has been seen across many other benchmarks. It’s a massive step forward that the issues in the original benchmarks were fixed, but these issues cause massive swings in performance, which can completely flip decisions and influence how agents are adjusted. In practice, if 30% of your tasks are broken, an agent that behaves correctly can score worse than one that happens to match the benchmark’s mistakes. We’ve seen teams fall into this trap, even with internal benchmarks & datasets: they see a number move, conclude a change worked (or didn’t), and steer accordingly. These systemic issues in datasets cost engineering teams many months chasing the wrong numbers and optimizing for the wrong things, leading them farther and farther away from reliable, production-ready agents. Your own dataset might be broken too At this point, an expected response may go something like this: “even if most public benchmarks are broken, I don’t use them. I built my own dataset from my own production data.” That’s definitely the right thing to do, but it’s also much harder to do well than it sounds, for three reasons we will briefly survey (it’s a really complex topic. More on this in another blog). Your dataset goes stale the moment you ship. Most teams build their dataset once, before production, and then either never update it, or painstakingly update it by hand But production data differs a lot from your original dataset. The same way a product accumulates bugs after release that no amount of pre-launch QA predicted. So a loop forms: you need observability into where your agent is messing up, you make changes to fix those failures, an [truncated for AI cost control]