Parametric CAD Bench

Back to CAD-Bench

May 13, 2026

•

Benchmark Release

Parametric CAD Bench: Can AI agents author editable CAD?

A benchmark for AI agents that design parametric 3D mechanical parts. The leaderboard lives at cadbench.ai; the artifacts behind it are the Harbor task suite gnucleus-ai/cad-bench and the Hugging Face results dataset gnucleus-ai/cad-gen-freecad-bench.

TL;DR

We introduce Parametric CAD Bench, a new evaluation for AI agents that measures the ability to author editable FreeCAD models from natural language. Unlike previous CAD-related benchmarks, we use a multi-step agentic loop and a strict "editability gate" (harmonic mean scoring) to ensure models produce functional engineering recipes, not just static 3D shapes. Early results show GPT-5.5 via Codex leading at 0.832, with a visible harness effect: swapping the driver while keeping the model fixed shifts scores by roughly 10% in either direction. Per-cell spend ranges from $3 to $170 across 100 trials each — the cost–quality frontier is wide enough that the top-scoring cell isn't always the best-value one.

What this measures

Modern AI coding agents — Claude Code, Codex, Gemini CLI, mini-swe-agent — can drive a Linux shell, edit files, run programs, and iterate on errors. We measure how well they can use those same tools to author an editable CAD design for a mechanical part described in plain English.

For each task in the suite, an agent receives:

A natural-language description of a part ("a round mounting flange with 4 bolt holes on an 87.3 mm bolt circle, 49.2 mm central bore, …")

A table of the part's key dimensions

The agent must then produce a script that, when executed by FreeCAD, creates a saved CAD document for that part. The document is graded against a held-back reference design for geometric correctness (does the resulting shape match the reference design) and consistency between the generated CAD and the provided spec. It is also gated by feature-based editability: for example, whether a CAD engineer can open the file and change a parameter such as bolt_circle_diameter through a feature or sketch.

CAD background, in two sentences: CAD software (Computer-Aided Design) is what engineers use to design physical parts. Parametric CAD means the design isn't a static 3D shape but a recipe — a sequence of named operations (sketch a circle, extrude it into a disc, cut a hole through the center, pattern that hole around the axis) whose inputs can be changed later. Editability is the whole point of CAD; a parametric flange where you can sweep outer_diameter from 80 mm to 120 mm and watch the bolt pattern adapt is worth a lot more than a fixed solid that happens to be 107 mm wide today.

There is also a major difference between a full CAD system and a geometry kernel or kernel-level modeling tool: FreeCAD vs. OCCT, SolidWorks/NX/Onshape vs. Parasolid, or AutoCAD vs. ACIS. A CAD system adds a rich layer of persistent engineering meaning, including named features, editable operations, constraints, parameters, stable face/edge references, and topological naming / persistent naming mechanisms. A pure geometry kernel mainly provides low-level geometric operations, while most real engineering workflows happen inside CAD systems, not directly on top of geometry kernels.

Why FreeCAD

FreeCAD is open source, fully scriptable from Python, and has a stable native format (.FCStd) that preserves the full feature history. Because it runs entirely offline, it's also straightforward to drop into a sandboxed container for automated evaluation. And it's a real CAD system that engineers use — the operations, constraints, and conventions match production workflows.

FreeCAD has two solid-modeling styles: the Part Workbench (CSG — booleans on primitives) and the Part Design Workbench (feature-based — sketches plus a parametric feature tree). We use Part Design because it captures engineering intent — parameter-driven operations (Pad, Pocket, Loft, Sweep, Pattern…) on top of sketches — and is how professional CAD (Catia, NX, Creo, SolidWorks, Onshape) actually works. That structure gives a much richer evaluation signal — right features used, parameters match, part still rebuilt when a dimension changes — which CSG loses once the booleans are applied.

Why this is different

Two CAD-AI benchmarks have appeared recently (more on both in How we compare to CadBench and BenchCAD). Both measure whether a single-shot prompt — text, image, or a 3D mesh — can get a model to output some kind of CAD program. They're impressive at scale — ~18,000 evaluation samples each — and they've quantified frontier models' shape-reproduction ability nicely.

We measure something different:

The agent has tools. A productive CAD-with-LLM session isn't a single prompt — it's a loop. Our agents run a shell, write the FreeCAD script, execute it, inspect errors, fix them, and try again, all inside a sandboxed container. The bench measures that loop end-to-end rather than just the first model output.

The output has to be editable, not just correct-shaped. Plenty of models can produce a 3D solid that matches a reference volume. Far fewer can produce a FreeCAD document whose internal structure (sketch → pad → pocket → polar pattern, with named driven dimensions) reflects how a CAD engineer would have built it. Our scorer rewards the latter — and because the two halves combine as a harmonic mean (see §"How scoring works"), a model that produces perfect geometry but zero named parameters scores 0, not 0.5.

The harness is a first-class variable. The 10 leaderboard cells cross four agent frameworks — claude-code, codex, gemini-cli, mini-swe-agent — with the major frontier models (Anthropic Opus / Sonnet / Haiku, OpenAI GPT-5.5, Google Gemini 3.1 Pro / Flash). Same model, different driver tells you whether the harness or the model is doing the work — a question this kind of leaderboard structure (borrowed from SWE-Bench and Terminal-Bench) is uniquely good at answering.

At a glance:

FeatureCadBenchBenchCADParametric CAD Bench

Primary goalGeometric accuracyIndustrial-standard coverageParametric / editable logic

MethodSingle-shot (vision)Single-shot (text / image)Multi-step agentic loop

Model writesCAD programCadQuery (Python) scriptFreeCAD-driving script

Graded artifactSTEP / meshCadQuery script (and the B-rep it produces)Native .FCStd with feature tree

Is the output CAD parametric?NoNo — script has parameters, generated CAD doesn'tYes — native feature tree + named dimensions

Key metricVolumetric IoUCode correctnessHarmonic mean of geometry × spec and cad consistency

How scoring works

Both sub-scores below are computed by the open-source gNucleus-AI/freecad-validator (v0.1.0) — the same validator binary is baked into each task image at /opt/grader/, so leaderboard scores and any local re-grade agree by construction. To run it yourself against an .FCStd:

pip install gnucleus-freecad-validator

Each trial produces two sub-scores in [0, 1]:

Sub-scoreWhat it checks

geometry_similarityDoes the shape match the reference design? Compared on solid count, volume, surface area, bounding box, and surface-type distribution.

cad_spec_consistencyFor each key parameter in the part spec (e.g. bolt_circle_diameter), does the saved CAD file contain a corresponding named, driven dimension within tolerance? A bolt pattern that's geometrically perfect but has the holes hard-coded as fixed positions loses points here.

The composite is the harmonic mean of the two:

Combined = 2 * (geometry_similarity * cad_spec_consistency) / (geometry_similarity + cad_spec_consistency)

This is the editability gate, expressed as arithmetic. A model that produces the correct shape but zero named parameters (cad_spec_consistency = 0) scores 0 combined, not 0.5 — there is no partial credit for "looks right but isn't editable." Symmetrically, a model that names every parameter but builds a wrong shape also scores 0. Both halves have to land for the trial to count.

The grading is a blind test. The agent receives only the natural-language part description and the parameter table — never the reference CAD file. The held-back reference geometry and the validator's tolerances live inside the task image at root-only paths the agent process can't read, so the agent is designing against the spec, not translating a reference it's been shown.

The matrix

The 10 leaderboard cells:

Agent-Model-ComboAgentModelWhat it tests

claude-code-opusclaude-codeClaude Opus 4.7Anthropic frontier

claude-code-sonnetclaude-codeClaude Sonnet 4.6Mid-tier Anthropic

claude-code-haikuclaude-codeClaude Haiku 4.5Cheapest Anthropic

codex-gpt5.5codexGPT-5.5OpenAI frontier

gemini-cli-progemini-cliGemini 3.1 ProGoogle frontier

gemini-cli-flashgemini-cliGemini 3.1 FlashGoogle cost floor

mini-swe-claude-opusmini-swe-agentClaude Opus 4.7Harness effect on Opus

mini-swe-gemini-promini-swe-agentGemini 3.1 ProHarness effect on Gemini Pro

mini-swe-gemini-flashmini-swe-agentGemini 3.1 FlashHarness effect on Gemini Flash

mini-swe-gpt5.5mini-swe-agentGPT-5.5Harness effect on GPT-5.5

Reading the leaderboard:

Vendor frontier comparison — filter to claude-code-opus, codex-gpt5.5, gemini-cli-pro to see which vendor's flagship model + native CLI does best.

Cost vs. quality — every row reports per-task USD spend. Sort by cost ascending and plot against score to see the cost-quality knee.

Harness effect — pair claude-code-opus with mini-swe-claude-opus (same model, two drivers) and the same for Gemini Pro, Gemini Flash, GPT-5.5. Wide gaps within a pair say the harness matters more than the model on this task type; narrow gaps say the opposite.

Preliminary results (v1)

All 10 leaderboard cells, sorted by mean composite score:

#CellMeanExceptionsCost

1codex-gpt5.50.8320$170.00

2mini-swe-gemini-pro0.7900$70.82

3mini-swe-gpt5.50.7440$42.35

4mini-swe-claude-opus0.7340$29.84

5gemini-cli-pro0.7290$51.28

6claude-code-opus0.6551$73.25

7claude-code-sonnet0.51822$96.34

8claude-code-haiku0.2848$24.67

9mini-swe-gemini-flash0.2410$3.00

10gemini-cli-flash0.1190$7.63

Total31$569.18

Headline findings:

GPT-5.5 via Codex tops the leaderboard at 0.832. The same model on a vendor-neutral harness (mini-swe-gpt5.5) scores 0.744 — a 0.088 gap that's this bench's clean harness-effect measurement on a single model. When the harness is well-tuned to its model (Codex was built for GPT-5.5), it lifts the score meaningfully.

The harness-effect runs in both directions. Codex is the only specialized vendor CLI that beats its generic counterpart; for the other three model families, the vendor-neutral mini-swe-agent driver wins:

ModelSpecialized CLImini-swe-agentΔ

GPT-5.5codex 0.8320.744+0.088 to Codex

Claude Opus 4.7claude-code 0.6550.734+0.079 to mini-swe

Gemini 3.1 Progemini-cli 0.7290.790+0.061 to mini-swe

Gemini 3.1 Flashgemini-cli 0.1190.241+0.122 to mini-swe

Anthropic's three-tier line lands at 0.655 / 0.518 / 0.284 for Opus / Sonnet / Haiku — a clean monotonic ladder by model tier. Sonnet's 22 exceptions split between two failure modes: 14 trials hit Sonnet 4.6's 32K-token output cap — the model tries to emit the entire FreeCAD script in a single assistant message and the response is truncated mid-code before any tool call lands. (This is an output-token limit, not a context-window limit; Sonnet's input context is 200K.) The remaining 8 exceptions are genuine iteration loops or wall-clock timeouts mid-tool-use. Opus runs clean (1 exception); Haiku trades scope for cost and lands cheap-but-low.

Reliability skews toward mini-swe-agent. The four mini-swe rows account for 0 exceptions across 400 trials. The vendor CLIs account for 31 exceptions across 600 trials, concentrated in claude-code-sonnet (22). The simpler "edit a file, run a command, look at the output" loop is more robust than the richer vendor drivers in this run.

Cost transparency. Each row carries token counts and per-trial USD spend

[truncated for AI cost control]