Frontier model evaluation for Physical AI

GPT-5.6 vs Claude Fable 5 for Physical AI, which performs best? - Blog | JuliaHub

Home

/

Blog

/

GPT-5.6 vs Claude Fable 5 for Physical AI, which performs best?

‹

›

Research & Innovation

GPT-5.6 vs Claude Fable 5 for Physical AI, which performs best?

Research & Innovation

GPT-5.6 vs Claude Fable 5 for Physical AI, which performs best?

Date Published

Jul 20, 2026

Contributors

Dr. Chris Rackauckas

Anas Abdelrehim

Anshul Singhvi

Share

Date Published

Jul 20, 2026

Contributors

Dr. Chris Rackauckas

Anas Abdelrehim

Anshul Singhvi

Share

We tested the latest frontier models gpt-5.6-terra in our agent, on five modeling and simulation problems of varying difficulty. Here are the results:

claude-fable-5

0.889

weighted score $9.60 / trial · 16.1 min

gpt-5.6-sol

0.814

weighted score $1.74 / trial · 13.4 min

gpt-5.6-terra

0.786

weighted score $1.25 / trial · 12.6 min

gpt-5.6-luna

0.727

weighted score $3.26 / trial · 25.0 min

A container combines a model, the Dyad harness, pinned configuration, and a sealed problem. The agent derives, compiles, simulates, and verifies a model before a grader compares it with sealed ground truth.

1 · the container

MODEL

gpt-5.6-terra

HARNESS

Dyad Agent

CONFIG

xhigh · 1M ctx · 128k

PROBLEM · SEALED

workspace + ground truth

↓

2 · the agent

▸ read problem▸ derive physics▸ write dyad/model.dyad▸ compile▸ simulate · verify

↓

3 · the grader

Simulate both, then compare

↓

4 · grade

trajectory match

0.96

Method · Schematic of the controlled evaluation pipeline; not a measured result.

Source · JuliaHub internal sealed evaluation · July 2026

A container combines a model, the Dyad harness, pinned configuration, and a sealed problem. The agent derives, compiles, simulates, and verifies a model before a grader compares it with sealed ground truth.

1 · the container

MODEL

gpt-5.6-terra

HARNESS

Dyad Agent

CONFIG

xhigh · 1M ctx · 128k

PROBLEM · SEALED

workspace + ground truth

↓

2 · the agent

▸ read problem▸ derive physics▸ write dyad/model.dyad▸ compile▸ simulate · verify

↓

3 · the grader

Simulate both, then compare

↓

4 · grade

trajectory match

0.96

Method · Schematic of the controlled evaluation pipeline; not a measured result.

Source · JuliaHub internal sealed evaluation · July 2026

A container combines a model, the Dyad harness, pinned configuration, and a sealed problem. The agent derives, compiles, simulates, and verifies a model before a grader compares it with sealed ground truth.

1 · the container

MODEL

gpt-5.6-terra

HARNESS

Dyad Agent

CONFIG

xhigh · 1M ctx · 128k

PROBLEM · SEALED

workspace + ground truth

↓

2 · the agent

▸ read problem▸ derive physics▸ write dyad/model.dyad▸ compile▸ simulate · verify

↓

3 · the grader

Simulate both, then compare

↓

4 · grade

trajectory match

0.96

Method · Schematic of the controlled evaluation pipeline; not a measured result.

Source · JuliaHub internal sealed evaluation · July 2026

Physical AI lives or dies on whether the modeled physics is correct. A model of an aircraft, a separation column or a charged particle can compile and run cleanly while the physics it encodes is impossible. Agentic AI makes this failure mode worse, because agents steer by feedback from tests, and the tests are often written by the same agent. In domains like web development or compilers that loop works well enough, since correct behavior is contained and checkable. In engineering, the real question is “does this match the real world?”, and that is much harder to close: the agent can pass every test it wrote while those tests rest on simplifications that don’t hold in the regime where the model will actually be used.

There is also a trust problem with the numbers that already exist. Nobody takes the model providers’ self-reported benchmarks at face value, and for good reason: the provider publishing the score is the same party with every incentive to tune for it. Our incentives point the other way. JuliaHub ships Dyad with multiple agent backends across vendors, and we win when our users get the best possible Dyad experience, whichever lab’s model delivers it. If one model is better at physical modeling, it is in our interest to find that out and recommend it. So we investigated: which one actually is?

To measure which frontier model handles that work best, we hold everything else still. The Dyad AI agent harness - purpose-built for modeling & simulation workflows - is pinned across every trial, along with the problems, the reasoning effort (xhigh), the context window (1M) and the token budget (128k). The only variable is the model: OpenAI’s GPT 5.6 family (terra, sol, luna) against Anthropic’s claude-fable-5. Three trials per model on each of the four core problems, one long-horizon trial each on the fifth: 52 graded runs in all.

01 · The problems

How Physical AI Evaluation Works.

We ground our exploration in a subset of our internal evals: five sealed problems drawn from the daily work of modeling & simulation. We order them by difficulty, defined in terms of the stages a correct solution must chain through, the careful details where a small slip is fatal, the easy shortcuts within reach, and the engineering work around the modeling itself - from parsing specs to repairing harnesses. We selected each for the same property: they admit solutions that compile and run green while being physically wrong. Grading therefore ignores the code and scores the full simulated trajectory against sealed ground truth. The hardest of the five is NASA’s HL-20 flight vehicle. An easier version of the problem is walked through in detail in this video; the study runs a harder, sealed variant.

02 · The scoreboard

What they scored.

We run every model through the full suite - three trials on each of P1–P4, one on the long-horizon P5 - and grade every trial against sealed ground truth. Per-problem scores fold into a single number through a difficulty-weighted average, the hardest weighing the most. Cost and time carry no weighting: average cost per trial and average wall-clock per trial each get a tab:

A toggleable comparison of difficulty-weighted score against cost or wall-clock time for four frontier models.

gpt-5.6-terraweighted score0.786

cost per trial$1.25

gpt-5.6-solweighted score0.814

cost per trial$1.74

gpt-5.6-lunaweighted score0.727

cost per trial$3.26

claude-fable-5weighted score0.889

cost per trial$9.60

Score against cost and time. Fable leads at a price: $9.60 a trial, $125 for the study — three to eight times the GPTs. Sol is the value story, second on merit at $1.74 a trial. Terra is cheapest and fastest. Luna trails on every axis.

Method · Difficulty-weighted score uses three trials for P1–P4 and one trial for P5. Cost and wall-clock time are unweighted means per trial.

Source · JuliaHub internal sealed evaluation · July 2026

A toggleable comparison of difficulty-weighted score against cost or wall-clock time for four frontier models.

gpt-5.6-terraweighted score0.786

cost per trial$1.25

gpt-5.6-solweighted score0.814

cost per trial$1.74

gpt-5.6-lunaweighted score0.727

cost per trial$3.26

claude-fable-5weighted score0.889

cost per trial$9.60

Score against cost and time. Fable leads at a price: $9.60 a trial, $125 for the study — three to eight times the GPTs. Sol is the value story, second on merit at $1.74 a trial. Terra is cheapest and fastest. Luna trails on every axis.

Method · Difficulty-weighted score uses three trials for P1–P4 and one trial for P5. Cost and wall-clock time are unweighted means per trial.

Source · JuliaHub internal sealed evaluation · July 2026

A toggleable comparison of difficulty-weighted score against cost or wall-clock time for four frontier models.

gpt-5.6-terraweighted score0.786

cost per trial$1.25

gpt-5.6-solweighted score0.814

cost per trial$1.74

gpt-5.6-lunaweighted score0.727

cost per trial$3.26

claude-fable-5weighted score0.889

cost per trial$9.60

Score against cost and time. Fable leads at a price: $9.60 a trial, $125 for the study — three to eight times the GPTs. Sol is the value story, second on merit at $1.74 a trial. Terra is cheapest and fastest. Luna trails on every axis.

Method · Difficulty-weighted score uses three trials for P1–P4 and one trial for P5. Cost and wall-clock time are unweighted means per trial.

Source · JuliaHub internal sealed evaluation · July 2026

The single number above has anatomy. Per problem, averaged across trials, the same three lenses:

A toggleable per-problem comparison of score, cost, and time for each model across five problems in difficulty order.

fablesolterraluna

P1easy

fable1.000

sol1.000

terra1.000

luna1.000

P2

fable1.000

sol1.000

terra1.000

luna0.667

P3

fable1.000

sol0.667

terra1.000

luna1.000

P4

fable1.000

sol1.000

terra0.667

luna1.000

P5very hard

fable0.690

sol0.660

terra0.590

luna0.383

Where the aggregate comes from. On score, the field holds the ceiling through the easier problems, each GPT slips once in the middle, and every line breaks downward at P5. On cost and time, P5 is where the money and hours go.

Method · P1–P4 values are means across three trials; P5 is one long-horizon trial per model.

Source · JuliaHub internal sealed evaluation · July 2026

A toggleable per-problem comparison of score, cost, and time for each model across five problems in difficulty order.

fablesolterraluna

P1easy

fable1.000

sol1.000

terra1.000

luna1.000

P2

fable1.000

sol1.000

terra1.000

luna0.667

P3

fable1.000

sol0.667

terra1.000

luna1.000

P4

fable1.000

sol1.000

terra0.667

luna1.000

P5very hard

fable0.690

sol0.660

terra0.590

luna0.383

Where the aggregate comes from. On score, the field holds the ceiling through the easier problems, each GPT slips once in the middle, and every line breaks downward at P5. On cost and time, P5 is where the money and hours go.

Method · P1–P4 values are means across three trials; P5 is one long-horizon trial per model.

Source · JuliaHub internal sealed evaluation · July 2026

A toggleable per-problem comparison of score, cost, and time for each model across five problems in difficulty order.

fablesolterraluna

P1easy

fable1.000

sol1.000

terra1.000

luna1.000

P2

fable1.000

sol1.000

terra1.000

luna0.667

P3

fable1.000

sol0.667

terra1.000

luna1.000

P4

fable1.000

sol1.000

terra0.667

luna1.000

P5very hard

fable0.690

sol0.660

terra0.590

luna0.383

Where the aggregate comes from. On score, the field holds the ceiling through the easier problems, each GPT slips once in the middle, and every line breaks downward at P5. On cost and time, P5 is where the money and hours go.

Method · P1–P4 values are means across three trials; P5 is one long-horizon trial per model.

Source · JuliaHub internal sealed evaluation · July 2026

03 · Work-style fingerprints

Each model has its own approach.

We analyze the transcripts behind every trial. Each tool call is bucketed by what the agent was doing at that moment - orienting in the problem, deriving and testing physics, editing the model, compiling, or hunting for something to copy - then counted per trial and averaged per model. The result is a work-style fingerprint:

Stacked bars compare how each model divides tool activity by call count or by time, with hover and focus tooltips.

orient & readderive & testedit the modelcompilehunt

orient & readderive & testedit the modelcompilehunt

terra

41 calls

sol

33 calls

luna

59 calls

fable

28 calls

102030405060

tool calls per trial (mean)

Two views of the same work. By calls, Luna is the busiest agent and Fable the leanest. By time, deriving and testing physics dominates every model's clock. Fable writes once; Terra spends more time editing; Luna alone spends time hunting.

Method · Tool calls are classified from transcripts, then averaged per trial

[truncated for AI cost control]