GraphRAG is overkill. This is what we do instead for knowledge graphs

Jai Juneja

Jul 20, 2026

👋 Hey, Jai here! I’m a software engineer and tech investor turned solo founder building a bootstrapped software company. I write about using AI to build faster, sell smarter, and run a whole company solo. New here? Join me:

Or if you don’t like my work, send me hate mail.

Every enterprise I’ve talked to at QX wants some variant of an AI “company brain” connected to their internal documents and systems. The belief is that this could unlock years of accumulated institutional knowledge and competitive advantage. Two years ago this warranted an exploratory pilot, but today it's table stakes.

The standard way to build this is called RAG, short for retrieval-augmented generation: break your company’s documents into passages, convert them into numbers that encode their meaning, and let an AI answer questions by retrieving the passages that most closely match a query. It’s simple, cheap, and demos well.

The problem is that RAG only performs well on a narrow set of questions - and they’re mostly the demo ones, not the ones where real value is created. That broken behaviour is the point of this post: why knowledge graphs (GraphRAG) were hyped as the ultimate fix, why the hype fizzled, and what we shipped instead.

Our approach has been shaped by what actually kills these projects in practice:

Information is unstructured and scattered across systems.

Retrieval quality has to survive scale, i.e. beyond a pilot.

Usage and uptake has to come before any cost or time commitment (as a rule, always lead with behavioural change).

Crucially, we wanted a system that our mid-market customers could switch on themselves, without consultants or setup workshops. That constraint heavily shaped our approach.

Three kinds of questions

When looking at the queries customers sent to their internal knowledge base, we identified three distinct shapes.

The first is a needle question. “What was the take rate in the 2017 valuation memo?” One fact, one document. Classic RAG is excellent here because searching by meaning finds that one passage reliably.

The second is a portfolio question. “Tell me everything we know about Acme Inc.” This is a broad, often multi-step question that needs every relevant document, not the top ten passages. RAG gets you a decent sample and silently drops the rest. You don’t know what you didn’t see, which for diligence or research work is the worst kind of failure.

The third is a counting question. “Which fintech companies have we evaluated?” or “Which contracts mention this supplier?” These need an exact list or tally across the entire corpus. Top-k search results cannot answer them. The system will confidently name the companies it happened to retrieve and skip the rest, and you’ll never know.

Most “AI brain” demos show needle questions. Most real business value hides in the other two.

Enter GraphRAG, the broken promise

In 2024, Microsoft Research published a technique called GraphRAG that took direct aim at this gap. Instead of just chopping documents into searchable passages, they proposed that an AI should parse through the whole corpus and build a knowledge graph out of it. Every company, person, and product becomes a node. Relationships between them become edges. Clusters of related entities get pre-written AI summaries, so the system can answer big-picture questions from the summaries instead of hunting through passages.

It’s a clever idea, and on a certain class of question it works. The research is consistent: graph-based approaches shine on questions that require connecting information across many documents, what the original paper called “global sensemaking”.

The hype cycle then did what hype cycles do, and GraphRAG became shorthand for “the serious way to do enterprise RAG”. That’s where the evidence stops cooperating.

What the research actually says

Three findings matter, and most of them come directly from Microsoft themselves.

The index is brutally expensive. Building the graph means running AI extraction and summarisation over everything before anyone asks a single question. Microsoft’s own follow-up work, LazyGraphRAG, states the cost plainly: a plain vector index costs about 0.1% of a full GraphRAG index. Three orders of magnitude, paid up front, on content nobody may ever query. The same work showed a lazy approach matching full GraphRAG’s answer quality on global questions at a small fraction of the query cost. Microsoft Research demonstrated that the expensive part of GraphRAG is mostly unnecessary. Their productised solution accelerator for GraphRAG was even archived in 2025.

The wins are narrow. A systematic evaluation published in 2025 (RAG vs. GraphRAG, arXiv:2502.11371) found what practitioners kept rediscovering: vanilla RAG wins on direct factual questions, graphs win on multi-hop and summary questions, and the best results come from routing between approaches rather than betting on either. Production case studies echo this. One well-documented deployment routes only a single-digit percentage of queries to its graph.

Entity resolution is the hardest part. For a graph to be useful, “Acme”, “ACME Holdings Inc.” and “Acme Corp” must become one node. Get that wrong in one direction and your graph fragments into disconnected shards. Get it wrong in the other direction and two different companies quietly merge into a single fictional one, and every answer built on top inherits the error. This problem is old and largely unsolved in the popular GraphRAG tooling. Meanwhile a graph database brings its own operational tax: another system to run, back up, secure, and keep consistent with your documents, multiplied by every tenant if you’re a SaaS product.

The philosophy behind GraphRAG is right: documents alone aren’t enough, and tracking the entities that your corpus talks about unlocks question types that ordinary search can’t address. The architecture most people associate with it, an eagerly built knowledge graph in a graph database, is overkill for the majority of real workloads.

What we built instead: graph-like RAG

We kept the parts with evidence behind them and dropped the rest. The result looks like a graph but runs on a regular database: it has entities, relationships, and rich metadata that evolve over time — but no Neo4j, no up-front graph build, and no ontology to custom-design.

Graph-like RAG: taking the best bits of knowledge graphs and scrapping the rest

Knowledge graph generated from a Vault on QX (FYI you can set up an account for free, upload some documents and visualise this yourself)

The breakdown is as follows:

Everything stays in ordinary infrastructure. Passages live in a search index. Entities live as plain database records: one record per company, person, product, project, event, or location, per client workspace. An entity record knows its name, its alternate spellings, which documents mention it, and which other entities it tends to appear alongside. Those “appears alongside” links are the graph. They’re just a list stored on each record, computed by counting co-occurrences. No graph query language and no new moving parts.

The ontology is fixed, small, and universal. Six types of things plus a handful of label fields (industry, category, topic) that adapt to whatever the documents contain. That fixed ontology is what makes the system self-serve. Custom ontologies are why traditional knowledge-graph projects begin with months of workshops. A pharma client and a private equity client get the same types, and the content of the graph adapts to their world automatically, because it’s extracted from their documents and evolves naturally as the corpus grows.

Entity resolution follows a clear waterfall. When a document mentions a name, we check a “phone book” of every spelling we’ve seen before. Exact hit: done, free. Miss: a similarity search proposes look-alike candidates for fractions of a cent. Only when there’s a genuinely ambiguous candidate does a small AI model get asked “are these the same real-world thing?”, and it’s instructed to say no when unsure. A wrong merge poisons everything downstream; a missed merge just leaves two entries that a recurring cleanup job can unite later. Every merge is reversible by design.

Summaries are lazy. Nothing gets an AI-written profile until someone actually asks about it. This is the lesson from Microsoft’s own LazyGraphRAG work, and it’s the difference between costs that scale with your corpus and costs that scale with your curiosity.

The AI routes between tools. The agent answering your question has plain search for needle questions, an exact “resolve” that pulls everything about one entity, an “expand” that walks to related entities (similar to “hopping” a graph database), and exact counting over the whole corpus for list questions. You can watch it chain these together. See the example below: we ask a broad “portfolio” question. It first uses facet to count entities (in this case companies) that get mentioned frequently in the context of food delivery, and then resolves each entity to dig deeper.

Does it actually work?

I’m obsessed with measurement, so before building any of this we constructed an evaluation harness: a corpus of thousands of real documents and a graded set of questions across all three question types, scored automatically. Every architectural decision above had to move a number or it didn’t ship. The short version is that it did: the entity layer produced a clear uplift on exactly the question classes plain RAG fails, portfolio and counting questions, while leaving needle questions at ceiling. The eval system deserves its own write-up, and it’ll get one in a future post.

The real trade-offs

It doesn’t do deep multi-hop reasoning. If your use case is “find the hidden path between these two entities four steps apart”, you want a real graph that can handle complex traversal.

Our links are co-occurrence, not typed relationships: the system knows that the company Acme and the person Jane Smith appear together constantly, but not that she’s the CEO. Our thesis is that, once the AI is supplied with the surrounding context for these two entities, it can gracefully deduce that Jane is the CEO without having to encode this in the graph itself.

Our conservative merging also means an occasional entity exists twice for a day until the cleanup job unifies it.

If your corpus is small, static, and your questions are research-grade sensemaking, full GraphRAG is a fine choice and the costs won’t hurt you. Our bet is different: for working companies with living document sets, the graph-like version captures most of the value at a fraction of the cost and none of the operational weight.

The part we care most about

Everything above runs self-serve. Connect Google Drive, SharePoint, Granola notes etc, and decide how frequently you want them to sync. The system extracts, resolves, links, and trues itself up on a schedule, and the costs of that upkeep scale with what changed, not with how much you’ve stored. Nobody designs a schema. Nobody labels anything. That was the constraint we started with, and it turned out to be the best design decision we made.

GraphRAG is a good idea wearing too much architecture. Take the idea. Leave the architecture.

This is how we approached it at QX Labs. If you’re wrestling with the same problems, or you think we’re wrong somewhere, I’d love to exchange notes.

Sources and further reading:

Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization (arXiv:2404.16130), the original Microsoft GraphRAG paper

Microsoft Research, LazyGraphRAG: Setting a new standard for quality and cost (Nov 2024), source of the 0.1% indexing cost and 700x query cost comparisons

Han et al., RAG vs. GraphRAG: A Systematic Evaluation and Key Insights (arXiv:2502.11371)

Anthropic, Introducing Contextual Retrieval (2024), the highest-evidence cheap retrieval upgrad

[truncated for AI cost control]