Beating a text-to-SQL benchmark: can you get better than plain Claude?

← All posts

Beating the BIRD-INTERACT benchmark: can you get much better than plain Claude by adding a semantic layer?

Egor Kraev · June 22, 2026

The best way to make a product good is to use it in a wide variety of cases. At Motley, we had already learned a lot from our customers and design partners while building SLayer, the open-source semantic layer for agents. The next natural place to look for that variety was text-to-SQL benchmarks.

TL;DR

With the Claude SDK and SLayer (the open-source semantic layer we developed), an agent reached a 75.3% pass rate on BIRD-INTERACT (a-interact, mini-interact), against the official best of 36.33% on the comparable Lite leaderboard.

The biggest lever was not the semantic layer, but the agent harness. The Claude SDK alone took a raw-SQL agent to 71.7%, while the PydanticAI-based setup yielded a much more modest improvement.

A substantial share of the benchmark’s gold answers are wrong, so we built an annotation agent and report results against both the original and the annotated answers.

SLayer still added an edge on top: 75.3% vs 71.7% against the original gold answers, and 83.7% vs 79.0% against the corrected (“annotated”) ones.

The SLayer ingestion ran fully autonomously for this run, so the numbers would probably be higher in a human-curated setup.

If you’re unfamiliar with SLayer, here’s what it does: sitting between the agent and the database, it holds the definitions the agent queries against: semantic models of your tables and fields (hand-written or auto-ingested from the schema), metric definitions like “net revenue”, and free-text knowledge attached as memories. Instead of writing raw SQL, the agent queries through structured JSON that SLayer compiles into SQL. A text-to-SQL benchmark exercises a large part of that job, answering business questions over an unfamiliar database with a knowledge base attached, so it was a natural way to stress-test SLayer.

So we ran one. With the Claude SDK and SLayer, an agent reached a 75.3% pass rate on BIRD-INTERACT (a-interact, mini-interact), compared to an official best of 36.33% on the comparable Lite leaderboard1. Boom. Headline number. The reality is more subtle, and the process to get there more interesting.

This post is the first in a series: here we cover how we got there and what the run taught us. Later posts go deeper on the individual lessons.

The benchmark maze

Text-to-SQL benchmarks are a fascinating, messy domain. Even if you just look at the BIRD family, it amounts to almost 20 different sub-benchmarks.

First there is the original BIRD, with its full and dev subsets. Then there is LiveSQLBench, with Base-Full, Base-Lite, Large-Full, and Large-Lite variants (and the occasional SQLite variant thrown in). Then there is BIRD-INTERACT with Full, Lite, and mini-interact variants, and each of those comes in c-interact and a-interact flavors, with separate leaderboards depending on whether the agent was budget-constrained. And then there is BIRD-CRITIC, with its own sub-benchmarks.

Laid out, the family looks like this:

BenchmarkVariantsNotable flavors

BIRD (original)full, devsingle-shot text-to-SQL

LiveSQLBenchBase-Full, Base-Lite, Large-Full, Large-Liteadditional SQLite variant

BIRD-INTERACTFull, Lite, mini-interactc-interact / a-interact, budget-constrained vs not

BIRD-CRITICown sub-benchmarksSQL debugging / critique

That is close to 20 distinct leaderboards before you have written a line of code.

We wanted to run SLayer in conditions close to how it would actually be used, which meant the Claude SDK for the agent and Opus as the LLM. That made a single full benchmark run cost on the order of $1000 for the smaller benchmarks, more for the larger ones. Running, and re-running to correct the issues that inevitably arose, across all the flavors was not an option. We had to settle on one to start with.

Why a-interact, mini-interact

We chose BIRD-INTERACT, a-interact flavor. What sets BIRD-INTERACT apart is that it tries to simulate a real working environment rather than a one-shot question. Each database is coupled with a hierarchical knowledge base, metadata files, and a function-driven user_sim simulator, so the agent can solicit clarifications, retrieve knowledge, and recover from execution errors on its own, with no human in the loop. The a-interact flavor takes this furthest: it is an open-ended agentic setting where the model decides for itself when to question the user simulator, when to explore the database, and how many turns to take before submitting a final answer. That is the closest of all the flavors to how agents actually operate in the real world, which is why we picked it.

We chose mini-interact because it is based on SQLite rather than Postgres, so it was less hassle to run, and it is smaller than the full version.

Put against the alternatives, here is why those two choices won:

Criterionc-interacta-interactFullLitemini-interact

Agent sets its own turn budgetnoyes–––

Provides a user_sim to questionyesyesyesyesyes

Database engine––PostgresPostgresSQLite

Number of query tasks––600195300

Closest to real agent usenoyes–––

There are no official leaderboard results for mini-interact yet. There are for the Lite version, which uses similar tasks but on Postgres databases rather than SQLite. The best result on a-interact there is 36.33%, using Opus-4.6 with a simple agent wrapper provided by the benchmark authors. That is the number we compare against, with the SQLite-versus-Postgres caveat noted up front.

The broken gold answers

The next challenge is that a substantial share of the gold answers the benchmark provides are simply wrong. This has been pointed out several times, for example here and here.

This is a real problem. To improve both SLayer and the agent harness, we need to distinguish cases where the agent actually went wrong from false negatives, where the agent’s answer was correct but evaluated against a wrong reference answer.

-- Task: "Show the accuracy ratio and confidence levels for each registration: -- registration ID, project ID, accuracy, error values, the calculated ratio (RAR), -- and the confidence level it translates to." -- Benchmark gold answer (the confidence-level branch): CASE WHEN rar > 1.5 AND sr.refmark LIKE '%Target%' THEN 'High Confidence' WHEN rar BETWEEN 1.0 AND 1.5 THEN 'Medium Confidence' ELSE 'Low Confidence' END AS confidence_level -- Why it is wrong: per KB #44, 'High Confidence' requires LogMethod to contain 'Target'. -- The gold matches against refmark instead, which only ever holds numeric codes -- ('40', '31', '25'), so the High-Confidence branch never fires and returns 0 rows. -- Matching on logmethod ('Target-based', 'Hybrid', 'Feature-based') gives the intended -- 50 High-Confidence rows.

So the first thing we built was an annotation agent. It went through each task and decided three things, given all the data the task made available to the agent:

whether the gold answer was the single correct answer,

whether it was compatible with the task at all,

and whether there were other answers that would be equally valid given the task statement.

These annotations are themselves probabilistic, so we report success rates two ways: implied by the original gold answers, and implied by the annotations. The annotated numbers are not gospel truth either, but they give a much better evaluation signal than relying on the original answers alone.

The experimental design

To separate the effect of the agent harness from the effect of using SLayer, we built two versions of every agent variant, as similar to each other as possible. One used SLayer to introspect the database and the associated knowledge base and to submit the answer. The other used raw SQL and the tools that shipped with the benchmark repo.

For the SLayer version, we used the standard SLayer schema ingestion to pre-populate the models, then deterministically created a field for each documented component of a JSON column in the original schema. The documentation for those was itself structured JSON, so the ingestion was trivial.

// Original schema: one documented JSON column, returns.return_details // (the benchmark documents each leaf of the JSON as structured text) { "column_meaning": "JSONB column grouping return reason, authorization, and shipping metadata.", "fields_meaning": { "fraud": { "risk_level": "VARCHAR(50). Fraud risk level for the return. NULL means not assessed." } } }

SLayer field created deterministically from it (one per documented JSON path)

• name: return_detailsfraudrisk_level

sql: JSON_EXTRACT(return_details, '$.fraud.risk_level') type: TEXT description: "Fraud risk level for the return. NULL means no fraud risk level assessed." meta: derived_from: json_col: return_details path: [fraud, risk_level]

Worth being clear about one thing here: for the benchmark, this entire ingestion ran fully autonomously and deterministically, with no human or LLM in the loop. That was deliberate, to keep the comparison clean and the result reproducible. In real use it is not how SLayer is meant to work. Human input at this stage to encode the knowledge base items would be massively additive. So treat the numbers below as a floor for what an automated setup gets you.

We also ingested the knowledge base items, raw pieces of text that document relevant concepts and can refer to each other, as SLayer memories. The way the knowledge base items cross-referenced each other matched the memories structure perfectly.

For the agent harness, we tried two setups:

PydanticAI with a custom sub-agent structure, spawning exploration and definition sub-agents for every concept needed.

Claude SDK with the appropriate set of tools.

We used Opus-4.7 throughout for the agent, and Sonnet for the user_sim.

What the benchmark taught SLayer

Because the benchmark contains such a varied set of queries, running it against the SLayer version was a great workout for SLayer itself.

It triggered a couple of edge-case bugs. It pointed out some types of queries the DSL could not yet represent at all, which we then added. And it gave us a collection of almost-but-not-quite-correct tool calls the agent made, which we used to enlarge our set of patterns that we auto-correct for during query parsing.

We are taking a unique approach in designing tools for SLayer: we account for potential failures by the agent and self-correct them when the intent is obvious. You can learn more in our earlier post, How forgiving are your MCP tools?

What the agent wrote (SQL-style aggregation -- appeared 265 times across the run):

count(*) sum(revenue)

What SLayer now auto-corrects it to (its native colon syntax):

*:count revenue:sum

This is the part that matters beyond the score. A benchmark with this much variety surfaces gaps a curated test suite never would.

The results

This is where we could boast of SLayer improving on the current leading model result by more than 2X: 75.3% with SLayer against the official 36.33% top-of-benchmark result.

But the more honest takeaway is that the choice of agent harness made the bigger impact on the pass rate.

We started with custom agents built using PydanticAI (basic loops with tools, where the tools included spawning task-specific sub-agents recursively). However, using claude_sdk yielded much higher pass rates (43% for PydanticAI-based vs 64% for claude_sdk, for a 53-task set we were using for that comparison), so we switched to claude_sdk throughout. Unfortunately, a comprehensive comparison of the two across the whole task set was not done for cost reasons.

The Claude SDK agents were a different story, with a significant improvement in performance overall. There, the SLayer-using agent performed marginally better than its raw-SQL twin.

HarnessVariantPass rate (vs original gold)Pass rate (vs annotated)

Benchmark authors’ wrapper (Lite, Opus-4.6)raw SQL36.33%n/a

Claude SDKraw SQL71.7%79

[truncated for AI cost control]