Skip to content
AI News HubLIVE
More
Summary only

Beating a text-to-SQL benchmark: can you get better than plain Claude?

Motley achieved a 75.3% pass rate on the BIRD-INTERACT benchmark using the Claude SDK and SLayer, far surpassing the official best of 36.33%. The largest improvement came from the agent harness (Claude SDK), with SLayer adding a modest edge. Many gold answers in the benchmark were found to be incorrect; an annotation agent built by the team yielded higher corrected pass rates.

SourceHacker News AIAuthor: yannranchere
Beating a text-to-SQL benchmark: can you get better than plain Claude?
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions

Summary only

Under the source content permission, this page provides an editorial summary, key points, and a link to the original article.

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • Combined Claude SDK and SLayer achieved 75.3% pass rate vs. 36.33% official best on BIRD-INTERACT.
  • The agent harness (Claude SDK) was the primary driver of improvement, not the semantic layer.
  • Many gold answers in the benchmark are wrong; annotation agent corrected them, resulting in 83.7% pass rate with SLayer.
  • SLayer ingestion was fully autonomous; human curation could further improve performance.

Highlights and analysis are generated automatically and may contain errors. Check the original source.

More recent stories on this topic · Research