Of Hammers and Nails: What AI Can and Cannot Do for a Data Analyst

Adam Cassar

May 21, 2026

Every few years, a new technology arrives with the same promise: this one will transform the organisation, eliminate the grunt work, and make whole categories of expensive people redundant. AI, and the large language models driving its current moment, is the latest. In data and analytics, the claims have been particularly bold — the well-prompted chatbot will soon replace the analyst, we are told. Having spent the past year rolling AI tooling out across a large organisation, the reality is more interesting, and more mixed, than that.

Where it genuinely helps is in writing code

Start with what works, because something genuinely does. AI tools have made writing code significantly faster. That matters more than it might sound. In teams that haven’t yet built mature data assets (data models), coding and data preparation is the job — easily 80 to 90 percent of what analysts actually spend their time on. Anything that speeds this up is a meaningful productivity gain.

Why does this matter so much? Because the single biggest factor separating a data team that delivers real value from one that simply costs money is the quality of its underlying data assets. Teams with clean, well-structured data models are dramatically more productive than those without. AI tooling can get organisations to that point faster, and that is a genuine advantage.

AI needs a lot of prep work on your data, and even then gets things wrong too often

The bigger claims for AI in analytics centre on its ability to answer ad hoc data questions — essentially, a tireless analyst available around the clock, queryable in plain English. The idea is appealing. The reality is not yet there.

The issue isn’t that AI can’t get the right answer. Sometimes it does, and well. The problem is consistency. Without very carefully built data architecture — clean models, good documentation, detailed instructions — the error rate is extremely high. And even with all of that in place, published results from companies that have done this work carefully show accuracy rates around 86 percent. That sounds reasonable until you think about what it means in practice - one in six answers is wrong. And you probably don’t know which ones. Would you be comfortable making a serious business decision knowing there was a 1/6 chance (at best) it was off? A human analyst getting things wrong that often would be a serious performance problem. The technology, held to the same standard, doesn’t pass.

It’s also worth noting that we already have a technology which answers business metric questions with near-perfect accuracy and never hallucinates. It’s called a dashboard. A well-built dashboard suite covers most of the questions stakeholders actually ask, reliably and cheaply. The case for replacing it with an AI query interface is weaker than the hype suggests.

Giving LLMs enough context to analyse metric changes well is often more laborious than doing the work yourself

This is arguably where AI disappoints most, because the use case sounds so compelling. The typical ask is something like: produce a report on how X performed and why.

Getting numbers is straightforward in a mature data setup — they’re already in your dashboards, or you can query them quickly. The hard part is the commentary: explaining why the numbers moved. In practice, that means sitting in meetings where teams discuss what drove their results, and chasing down the people who actually know. That knowledge lives in people’s heads, not in databases.

For AI to write that commentary usefully, someone has to first write down all that context in a form the model can use. And the effort of doing that turns out to be roughly similar to just writing the report yourself. Without automated ways of capturing that context the time saved is surprisingly small. Things like integrations into team Slack channels and structured meeting notes can help, but it’s both imperfect and still suffers from the hallucination problem.

What you often get instead is work that’s slightly better rather than significantly faster. The models do often spot things that humans miss, even if the overall error rate is meaningful, and it handles the mechanical parts of the drafting well. Those are genuine benefits. But the transformative reduction in analyst workload that has been predicted? That hasn’t materialised — at least not yet.

The bottom line

AI tooling has made real contributions to data work: faster code writing, quicker data asset development, and a useful drafting assistant for analytical writing. These gains are worth having.

What it hasn’t done is replace the core of what good analysis requires — judgement, context, institutional knowledge, and the ability to tell a correct number from a plausible-looking wrong one. The analyst job apocalypse confidently predicted by people selling AI products has not arrived. Most experienced analysts probably knew it wouldn’t.

The right approach is neither uncritical enthusiasm nor dismissiveness. Use the tools where they help. Be clear-eyed about where they don’t. The teams that get this right will outperform those that don’t — whether they’re over-investing in AI or ignoring it entirely.