Benchmarks Are Dead (For Us)

Poetiq | Benchmarks Are Dead (for us)

get in touch

[email protected]

join our team

[email protected]

Our RSI loop now fully automatically builds its own state-of-the-art harness for any task it encounters.

July 15, 2026

Mainstream AI research is fixated on a capital-intensive strategy — intelligence via incremental weight updates, costing billions in compute. At Poetiq, we believe that intelligence does not need to live solely within model weights. Instead, we treat the LLM as a single component of a larger, self-improving reasoning architecture. This self-optimizing architecture leads to a Recursive Self-Improvement (RSI) loop that is designed to automatically construct entire harnesses (consisting of code, prompts, tools, and search strategies) for any benchmark it encounters.

At Poetiq, we're building a Metasystem capable of autonomously generating the harnesses and auxiliary data required to solve any challenge. Setting state of the art (SOTA) results on newly introduced benchmarks is now a routine procedure for us. However, benchmarks only serve as proxies for real world challenges; their diagnostic utility plummets once our system automatically masters them. Conventional, static, benchmark frameworks are inadequate for evaluating systems with true RSI capabilities.

The Benefits of our RSI Loop

Autonomous SOTA Generation: With zero human intervention, our Metasystem independently built harnesses that set SOTA milestones across six diverse benchmarks — including long-context retrieval, agentic tool use, complex system planning, and competitive mathematics. This includes outperforming a SOTA benchmark result from Muse Spark 1.1 within 48 hours of publication.

Disruptive Performance: We frequently secure these SOTA outcomes without using the benchmark's leading model. In multiple instances, our system overtook existing benchmarks held by Anthropic's Claude Fable 5 using a previous generation model.

A Model-Agnostic, Compounding Moat: By decoupling reasoning mechanics from underlying weights, our design is completely model-agnostic. As proprietary and open-source models advance, our Metasystem utilizes them as tools, continuously compounding their value.

The Path That Brought Us Here

In the past year, we've been deliberate about which benchmarks we attempt. We picked our initial three to match critical categories of tasks for LLMs:

Reasoning — synthesizing provided information in inventive ways. ARC-AGI (followup) is the premier test of this.

Retrieval — testing the limits of the breadth of knowledge embedded in a model's weights. Humanity's Last Exam audits this across a vast spectrum of disciplines.

Coding — the most pervasive commercial application of AI, melding reasoning and retrieval with specialized procedural logic. LiveCodeBench Pro tests coding ability while minimizing the risk of model memorization of answers.

We set SOTA on all three with no fine-tuning of weights and no privileged access. Across the board, we see gains on all proprietary and open-source models. With each test, our Metasystem constructed and optimized increasingly large portions of its own harness, widening its capabilities through RSI.

The loop is now good enough that it no longer needs us to do more than pick the benchmark. No hand tuning or customization needed.

The Results: Brand New SOTA Benchmarks

In the following table, we demonstrate the efficacy of our automatic Metasystem on six highly complex benchmarks. Each benchmark was chosen to target new and diverse tasks of types that our system had not previously seen. The same Metasystem built every harness automatically. It designed the code, prompts, and hyperparameters. For half the benchmarks, SOTA was achieved even with one-generation older models than the current SOTA holder.

Each harness, created by Poetiq's Metasystem, improved across all models (both proprietary and open-source). Below, we give our results and the models that our system used to reach it.1

(Note: While we did not use Anthropic's Claude Fable 5 as our underlying model, we routinely outperformed its capabilities.)

Benchmark Category Underlying models used by Poetiq Poetiq Prev. SOTA Prev. model

ArXivMath Competition math GPT-5.5 89.2 87.5* Claude Fable 5

SciCode Scientific coding Gemini 3.1 Pro 61.5 60.2 Claude Fable 5

Haladir — Challenge Set Long horizon planning Gemini 3.1 Pro 0.47 0.28 Claude Fable 5

Haladir — Full Set Long horizon planning Gemini 3.1 Pro 0.69 0.33‡ Claude Opus 4.8

MCP-Atlas Agentic tool use Muse Spark 1.1 & Gemini 3.5 Flash 89.8 88.1 Muse Spark 1.1

MRCR v2 Long-context Gemini 3.1 Flash-Lite 99.26 97.3 GPT-5.4

Toolathlon Agentic tool use Gemini 3.5-Flash 59.26 56.5† Gemini 3.5 Flash

Scores are percentages except Haladir, whose native score is 0–1. * Model released after our results were reached. † Prior best on the standard, unmodified benchmark. Per Anthropic's own note, its reported Claude Fable 5 (61.7) and Opus 4.8 (59.9) run “~3 points above a strictly upstream-equivalent harness” (Fable 5 System Card, §8.17.8). ‡ Mean of ending budget ÷ initial budget over 200 environments.

ArXivMath — leapfrogging on the same base model: ArXivMath is a contamination-resistant math benchmark: its problems are drawn monthly from fresh ArXiv papers published after the models' training cut-offs. It probes genuine mathematical reasoning — multi-step derivation and proof — rather than recall. It is one of the cleanest available measures of generalized reasoning.

Result: Poetiq took GPT-5.5 from 77.5% to 89.2%, and Gemini 3.1 Pro from 51.7% to 68.3%. Our GPT-5.5 result also exceeds the previous SOTA of 87.5% posted by Claude Fable 5 (max) — a model released after our results were achieved.

SciCode — clearing the SOTA with a smaller model: SciCode asks models to implement research-grade scientific code from a natural-language problem statement. These problems are derived from complex domains such as physics, chemistry, biology, and beyond. The benchmark rewards the synthesis of domain knowledge with precise procedural logic, not competitive-programming tricks. It measures whether AI can do the kind of coding that moves science forward.

Result: We achieved 61.5% with Gemini 3.1 Pro, overtaking Claude Fable 5's recent SOTA of 60.2%.

Haladir: Operations Research — more than double the previous state of the art: In the Haladir benchmark an agent is tasked with running a simulated factory: 200 scenarios across 10 synthetic domains — commercial baking, CNC machining, brewing, textiles manufacturing and more. In each scenario, the agent must build stations, hire workers, order materials, and ship product orders with operational constraints (limited budget, harsh deadlines). It is scored on the ratio of ending budget to initial budget. Because winning means holding many interacting constraints at once (station health, ingredient spoilage, worker fatigue, deadlines) it is a model of real operational work and the difficulties of long-horizon planning.

Result: On the full 200-environment set, Poetiq lifted the score from a prior best of 0.326 (Opus 4.8) to 0.688 with Gemini 3.1 Pro — more than double the previous state of the art. The lift holds across nearly every domain.

Additionally, Haladir released results on a harder 20-scenario subsample. On this subsample, Haladir reported results for Claude Fable 5 at 0.28. Using our same expert, we reached 0.470 — 1.7× that of Fable.

MCP-Atlas — SOTA on tool use: MCP-Atlas evaluates real-world tool use over the Model Context Protocol. Across simulated servers and real APIs, a model must discover the right tools, call them correctly, recover from errors, and synthesize the results. It measures whether a model can act independently, not just answer. This is the capability that separates an agent from a chatbot.

Result: We reached 89.8% on the 500-problem public set using a combination of Muse Spark 1.1 and Gemini 3.5 Flash, exceeding the prior best of 88.1%.

MRCR v2 — reading two million tokens with a model that isn't on the leaderboard: MRCR v2 is a “needle-in-a-haystack” benchmark: the model must find and correctly disambiguate eight nearly-identical passages buried in roughly 2M tokens of context. It measures whether a model can genuinely use its whole context window. As context windows keep growing, it's the test that shows whether those extra tokens in context are truly usable.

Result: We hit 99.26% – past the SOTA of 97.3% set by GPT-5.4 — using Gemini 3.1 Flash-Lite, a small, inexpensive model whose base score doesn't even crack the leaderboard (even Gemini 3.1 Pro manages only 84.9%).

Toolathlon — SOTA on the real, unmodified set: Toolathlon is a suite of 108 real-world tool-use tasks spanning office productivity, e-commerce, data analysis and web research. These tasks are graded by execution-based checkers that verify the actual side effects each task should produce. With 600+ tools across dozens of applications and roughly twenty turns per task, it stresses long-horizon orchestration across many apps. It is a demanding measure of whether an agent can reliably finish real multi-application workflows.

Result: We reached 59.26% with Gemini 3.5 Flash — the best result on the benchmark as of this publication, past Gemini 3.5 Flash's 56.5% (without Poetiq) and every off-the-shelf frontier model (GPT-5.5: 55.6%, Sonnet 5: 54.3%, Gemini 3.1 Pro: 48.8%).

The only higher figures you may have seen — Claude Fable 5's 61.7% and Opus 4.8's 59.9% — come from Anthropic's modified Toolathlon set along with Anthropic's own caveat that its reported numbers “consistently run ~3 points above a strictly upstream-equivalent harness”2. On the clean, unmodified benchmark, we hold the top spot.

The Future of Benchmarks

Every result above shares one property: we didn't do the work. We didn't read the benchmark, design a prompt, or hand-build a scaffold. The RSI loop ingested the benchmark and produced a custom, task-specific harness on its own — across retrieval, math, coding, and tool-use alike.

And that's been our goal since we started Poetiq. The goal was not to create a harness to beat a specific problem but to create a Metasystem that can automatically create harnesses, data, and procedures to solve any problem. We treat the LLM as a single component of a larger reasoning system (code, prompts, exploration/exploitation strategies, and more) and what improves in each iteration is the system, not necessarily the model's weights.

This reframing is what lets our RSI loop run quickly and cheaply today while still compounding its benefits. It also frees us from betting on any single model: we use models as interchangeable tools. The Poetiq Metasystem is a self-optimizing optimizer. Each problem, task, and dataset it tackles helps it to optimize itself, building itself into a more and more powerful optimizer.

Of course, the statement “benchmarks are dead” is a bit tongue-in-cheek. An important question, in the light of RSI systems like ours which are already constantly self-improving, is how to find gaps in the current model's knowledge and enable further improvement. Perhaps unsurprisingly, we've turned to our own systems for the answer. Because the Poetiq Metasystem maps the capabilities of the underlying models it uses, we can also task the system to find highly challenging datasets along the model's jagged frontiers.

Beyond just discovering such static datasets, we can dynamically generate benchmarks that combine information and abilities that no single model has — a living benchmark that can't be trained against. So, perhaps it is more appropriate to say: the static benchmark is dead; long live the living benchmark.

What's Next for Poetiq: Continuous RSI

We're turning the RSI loop on and leaving it on. Concretely, that means:

Ever-improving systems fueled by solving real problems for enterprise partners: By design, we created the Poetiq Metasystem to improve throug

[truncated for AI cost control]