Comparing Context Retrieval Approaches for AI Code Review

Meerkat Careers | Comparing Context Retrieval Approaches for AI Code Review

Blog

Comparing Context Retrieval Approaches for AI Code Review

Back To Blog

Overview

At Compare The Market, we have an internal AI tool that automatically reviews merge requests. The goal is to speed up the time it takes for developers to receive feedback on their code and increase MR throughput across the organisation. Developers get an intelligent first-pass review within minutes of opening an MR.

It works well, but we wanted it to work better. The reviewer was leaning on the side of conservatism, where we often saw false positives in bug detection due to the limitations of reviewing code changes in isolation. We wanted to provide the reviewer with an understanding of how the change sits within the broader system. For example, a deleted function might seem like dead code, unless you know it’s called dynamically from another service.

At this point we faced a fundamental architectural decision: How should the agent retrieve context about the codebase?

We had two main options:

GKG (GitLab Knowledge Graph): A code analysis engine that uses Tree-sitter AST parsing (via gitlab-code-parser) to build a structured knowledge graph of code entities and relationships, stored in a Kuzu graph database. Enabling precise queries like “find all callers of this function” or “show me the class hierarchy” RAG (Retrieval-Augmented Generation): A vector similarity search approach that chunks code, creates embeddings, and retrieves semantically similar code snippets.

We chose GKG based on intuition – our hypothesis was that code review requires structural understanding of code relationships, not just semantic similarity. When reviewing a change to a function, you need to know what calls it, what it calls, and how it fits into the broader architecture. RAG excels at finding “similar” code, but similarity isn’t the same as relevance for code review.

This article validates that intuition. Through rigorous evaluation using MLflow on Databricks, we compared four approaches and found that GKG outperforms RAG on the metrics that matter most for code review quality. The data confirms our architectural decision was correct.

1. The 4 approaches

We evaluated four distinct configurations:

1. GKG Integration

What is GKG?

Last year, GitLab introduced a beta version of an MCP (Model Context Protocol) server called the GitLab Knowledge Graph (GKG). The API indexes the repository and builds a structured, queryable representation of the codebase. It maps dependencies onto nodes in a graph, understands function definitions and their usage, traces inheritance hierarchies, and captures cross-references between modules.

The result is a semantic map of your code – not just a list of files, but a web of relationships. Through the tools provided by the MCP server, AI agents can query this graph in real time:

“Where is this function called?” “What classes inherit from this interface?” “What would be affected if I changed this method signature?”

Our Sidecar Integration Because GKG was still in beta and not yet available as a native GitLab CI/CD feature, we built a separate sidecar service – a lightweight Docker container that wraps the official GKG binary and runs alongside our reviewer in the CI pipeline.

The Workflow Index – When a merge request pipeline kicks off, the sidecar container mounts the project source and indexes the full codebase, building the knowledge graph from scratch. Serve – Once indexed, it starts the GKG MCP server on a local port, exposing a set of tool calls. Query – Our AI reviewer connects to the MCP server and uses these tools as part of its review workflow.

How the Knowledge Graph Works GKG builds a symbol graph — a structured representation of your codebase where nodes represent code entities (classes, functions, variables) and edges represent relationships (calls, inherits, imports).

The Indexed Project Graph When GKG indexes a repository, it creates an interactive graph visualisation showing the entire codebase structure. Here’s what the indexed graph might look like for an example project:

Each node type represents a different code entity:

Orange (Directory) – Folder structure of the repository. Green (File) – Individual source files. Purple (Definition) – Classes, functions, and methods defined in the code. Blue (Imported Symbol) – External dependencies and imports.

The edges (lines) show relationships: which files contain which definitions, which functions call other functions, and which modules import which symbols.

Example: Symbol Graph Structure Consider a simple UserService class. GKG maps it as a graph showing the class, its methods, and all the files that call those methods:

Example: Querying the Graph When the AI reviewer needs to understand the impact of a change, it queries GKG. For example, if someone modifies validate_input(), the agent asks: “Who calls this function?”

This precise information allows the reviewer to assess whether a change to validate_input() could break any of these three callers – something impossible with just the diff alone.

How GKG Improves Review Quality

Now the reviewer is able to utilise GKG to query the knowledge graph to verify the initial concerns our discovery agent identifies against the wider codebase. For example, if a change looks like it might break a contract, the agent traces the dependency chain to verify before reporting it as an issue.

By verifying initial concerns against the actual structure of the codebase, the reviewer produces more accurate, more consistent feedback, and critically, fewer false positives in bug detection.

GKG Architecture Overview

1. RAG Integration

Our RAG implementation uses LlamaIndex for intelligent code chunking and OpenAI embeddings for vector similarity search.

Phase 1: Indexing Pipeline

The indexing pipeline:

Source Files: Scan repository for code files (.py, .js, .ts, etc.). CodeSplitter: LlamaIndex AST-aware chunking. Code Chunks: Semantic units (functions, classes) with overlap. Embeddings: OpenAI text-embedding-3-small. FAISS: Vector store for fast similarity search.

Phase 2: Retrieval Pipeline

LlamaIndex CodeSplitter: Graph-First Approach

The key component of our RAG implementation is the CodeSplitter from LlamaIndex. Unlike naive text chunking, CodeSplitter uses a graph-first approach: it first builds an AST (Abstract Syntax Tree) graph of the code, then uses this structural understanding to create semantically meaningful chunks.

Step 1: Build the AST Graph

Before any chunking happens, Tree-sitter parses the source code into a hierarchical graph structure. This graph represents the syntactic structure of the code – functions, classes, methods, and their relationships.

Step 2: Traverse Graph to Create Chunks

Once the AST graph is built, CodeSplitter traverses it to identify semantic boundaries. Instead of blindly splitting at line 40, it finds natural break points:

Function boundaries: Each function becomes a chunk (or multiple if large). Class boundaries: Class definitions with their methods. Logical groupings: Related code stays together.

Why RAG May Struggle with Code Review

The Fundamental Limitation: RAG relies on semantic similarity in vector space, which works well for natural language but has inherent limitations for code:

1. No Symbol Resolution

RAG cannot distinguish between a function definition and a function call with the same name. It treats def process_data() and process_data() as semantically similar without understanding the relationship.

1. No Reference Tracking

When reviewing a change to a function, RAG cannot reliably find all callers of that function. It may return semantically similar but unrelated code.

1. Chunk Boundary Issues

Even with AST-aware chunking, important context may be split across chunks. A function’s signature might be in one chunk while its implementation is in another.

1. Precision vs Recall Trade-off

RAG optimises for semantic similarity, which may surface “related” code that isn’t actually relevant to the review. This can overwhelm the LLM with noise.

1. Evaluation Setup

The Challenge: Measuring Quality Without a Single Correct Answer

Evaluating a generative AI reviewer is fundamentally different from evaluating a classifier or a search engine. A code review has no canonical correct output – two expert engineers reviewing the same merge request will write different, equally valid feedback. This means standard accuracy metrics don’t apply, and simple text comparison is meaningless.

The challenge compounds when you consider that quality is multidimensional. A review might identify the right risks but overstate severity. It might be perfectly calibrated in scoring but miss the one critical issue. It might surface the correct concern but point to the wrong line of code. Each of these failure modes matters independently, and optimising for one can actively degrade another. This is what makes rigorous AI evaluation hard: the outputs are open-ended, non-deterministic, and semantically rich.

Ground Truth: A Curated Reference Dataset of 79 MRs

To evaluate reliably, we need a fixed reference point. We constructed a golden dataset of 79 real merge requests from our internal repositories, a structured benchmark where each entry carries expert-annotated ground truth across multiple dimensions:

Expected summary points – the key changes and their implications. Expected issues – risks, bugs, and concerns that a thorough reviewer should surface. Expected inline comments – specific defects, tied to exact file locations and code lines. Expected score range – the range of severity scores a calibrated human reviewer would assign.

Constructing this dataset required deliberate effort. The MRs represent real production complexity, covering different codebases, change sizes, and types, rather than easy cases favoring any approach. Each entry was annotated with a consistent rubric defining a thorough, well-calibrated review: identifying the right issues, flagging the correct lines, and assigning a severity score reflecting genuine impact.

Scoring: Core Quality Dimensions

All four approaches (Baseline, GKG, RAG, GKG+RAG) were evaluated on the same 79 entries. Each generated review was assessed across five core quality dimensions:

Coverage – How many expected summary points, issues, and inline comments appear in the output? A review missing a critical bug scores low here regardless of addressing other issues.

Precision – Of everything the reviewer surfaced, how much was actually warranted? This penalizes over-generation: a review flagging 20 issues when 5 were expected may have good coverage but poor precision, indicating a noisy, unreliable reviewer.

Inline comment location accuracy – Code review is inherently spatial. It’s not sufficient to identify a problem in prose. It needs to be attached to the correct file and anchored to the relevant code change. We verified this separately from semantic correctness.

Score calibration – The reviewer assigns a severity score from 0 – 10. We measured whether that score fell within the range a human would consider reasonable. This matters for developer trust: a reviewer that consistently over or under penalises loses credibility quickly.

Structural validity – a schema check ensuring required fields are present and well-formed (e.g., non-empty summary and score, inline comments include required fields). All approaches consistently passed this check in our runs.

LLM-as-Judge: Semantic Evaluation at Scale

For nuanced criteria, such as whether a generated summary “covers the same point” as a reference, text matching fails. “The function signature changed” and “the method contract was modified” are semantically equivalent but lexically distant.

We used an LLM as an automated judge. Given the reference expectations and generated output, the judge counted how many expec

[truncated for AI cost control]