GPT-5.6, Fable 5, and Grok 4.5 rebuild Basecamp from the same spec

GPT-5.6 Sol arrived with OpenAI claiming stronger frontend engineering and greater token efficiency than GPT-5.5, while Anthropic released Fable 5 at a dramatically higher price than the other models. SpaceXAI launched Grok 4.5 in the same period as a comparatively inexpensive, Opus-class model that, in theory, should be extremely capable.

I wanted a way to test those claims through real-world use. I chose one sufficiently complex software problem and dissected it across 20 dimensions to see where the models excel, where they fail, and where their performance is nearly indistinguishable.

Benchmark setup and results

Fable 5 frontend prototype showing global search in its Basecamp implementation

I built Basecamp Bench to measure this. You can read the full report here. Agents start in the same greenfield workspace with AGENTS.md, DESIGN.md, INIT.md, and Basecamp 5 reference material. They then build a frontend from mocks (screenshots) and design tokens, and a backend implementing the canonical API contract, revealing their engineering choices and tradeoffs.

Fable 5 won both tracks. Grok 4.5 delivered the strongest speed and cost tradeoff, completing both builds in 37 minutes for $9.30, but with significant polish gaps.

Grok 4.5 frontend prototype showing the sample Basecamp project

ModelFrontendBackendTotal timeTotal cost

Fable 57.5788.3922:06:40$85.87

Sonnet 56.9827.2431:27:09$36.23

Grok 4.56.3847.27836:48$9.30

GPT-5.6 Sol5.7657.31059:48$15.13

GPT-5.55.6707.08444:14$10.94

Agent separation

Backend scores cluster tightly because most models register the complete route surface, with invariant enforcement, contract-correct responses, and honest failure handling separating them. Frontend scores vary more as models trade breadth for depth and struggle with polish: the spacing, icons, micro-interactions, and transitions that make an interface feel finished.

Backend benchmark scores across evaluation dimensions

Fable 5 follows the specification most closely, connecting every surface and endpoint to the same product model. Sonnet prioritizes working, persistent workflows, GPT-5.5 prioritizes polish (though not particularly well) while leaving core controls inert, and GPT-5.6 Sol pairs a disciplined backend with a shallow frontend.

Frontend benchmark scores across evaluation dimensions

Grok 4.5 delivers the strongest balance of speed, cost, and functional depth. Its higher frontend score than Sol's reflects greater functional completeness despite the layout issues visible in the screenshot above, and it nearly matches Sol on the backend.

For me, Fable 5 is within a few percentage points of the real Basecamp frontend implementation, with very minor but detailed adjustments that a talented design engineer could finish.

The real Basecamp frontend showing its sample podcast project

The gap between Fable 5 and the other models is striking, especially given how heavily OpenAI1, Anthropic2, and SpaceXAI3 tout them for frontend work.

The final 10% of a project requires as much work as the first 90%.4 Each point on Basecamp Bench is harder-won than the last. Moving from eight to nine requires far more depth and complexity than moving from five to six.

Browse each frontend submission in the repository ↓

Fable 5

Sonnet 5

GPT-5.5

GPT-5.6 Sol

Grok 4.5

Run variability

Score variability across five Sonnet 5 and GPT-5.6 Sol runs

I ran Sonnet 5 and GPT-5.6 Sol five additional times on each track to measure run-to-run variation. Stronger runs turned more of the requested product into working, persistent behavior and handled API contracts and validation more carefully, while weaker runs often looked similarly complete at a glance but concealed inert controls, brittle state, and shallow backend behavior. The spread reveals each model's latent capability range and the risk of building from an unlucky draw.

Model / trackMedianObserved rangeMedian costBest of 5Δ vs. published

Sonnet 5 FE6.5856.178–6.922$13.726.922−0.060

GPT-5.6 Sol FE6.0265.780–6.489$6.476.489+0.724

Sonnet 5 BE7.4607.346–7.743$24.047.743+0.500

GPT-5.6 Sol BE6.9956.675–7.313$11.407.313+0.003

Full report and other agents

I used the new GPT Sites feature to publish the evaluation report, where you can explore the methodology, dimension-level scores, charts, model-by-model commentary, and specific failure modes.

The GitHub repo contains the runner, prompts, rubrics, versioned contracts, reference material, baseline runs, evaluator reports, leaderboards, and provenance manifests needed to inspect or reproduce the benchmark.

Building the project with GPT-5.6 Sol consumed 1.8 billion tokens and $2,867.52 in model usage.

I attempted to test ZAI's GLM 5.2 using the Pi harness, but the model is unable to complete the benchmark as written.5 I also attempted to test Gemini Flash 3.5 in AGY, but experienced tool call failures across five runs.6 I successfully tested Gemini Pro 3.1, but both submissions were largely unfinished, scoring 3.2 on the frontend and 3.9 on the backend. The model appeared unable to sustain agentic work across the benchmark's long time horizon, placing it in a different class from the other agents.

I will add Gemini 3.5 Pro when it releases in the coming weeks. If the rumors are true and GPT-6 lands later this summer, I will add it to the benchmark as well.

If you want to add another harness or benchmark run, submit a PR or send me a note.

This post was dictated to GPT-5.6 Sol using ChatGPT.

Fable 5, Grok 4.5, and GPT-5.6 Sol were used to create the evaluation code. The report content was dictated to both Fable 5 and GPT-5.6 Sol. Fable 5 created the final report, with Sol correcting its writing mistakes and improving its clarity.

Footnotes

OpenAI, GPT-5.6 and GPT-5.6 Sol model documentation. ↩

Anthropic, Introducing Claude Sonnet 5. ↩

SpaceXAI, Introducing Grok 4.5. ↩

“The first 90% of the code accounts for the first 90% of the development time. The remaining 10% of the code accounts for the other 90% of the development time.” — Tom Cargill ↩

Both benchmark tracks ask the agent to produce a single-file implementation, and GLM 5.2 repeatedly broke on that constraint. I manually overrode the harness to allow multiple files and tried having the model produce them in stages, but the agent still could not complete either run. ↩

AGY lost background-task state in all runs, then misclassified completed work as failed. Missing task IDs broke polling on the backend, while missing task logs caused an invalid tool call and runaway generation on the frontend. ↩