2026-05-27原文

SPEAR: Code-Augmented Agentic Prompt Optimization

SPEAR (Sandboxed Prompt Engineer with Active Roll-back) is a free-form agentic optimizer that ports the code-as-action paradigm to automatic prompt engineering. It features four tools—evaluate, python, set_prompt, finish—and decides autonomously how to use them. The key innovation is a Python sandbox for structural error analysis on evaluation DataFrames. Two guardrails (auto-rollback and guard metric floor) ensure monotonic improvement. Evaluated on three industrial LLM-as-judge suites (13 tasks) plus 7 BBH tasks and GSM8K, SPEAR wins all industrial tasks on primary metrics and achieves 0.938 accuracy on BBH-7. Ablations show the Python tool is the largest single lever.

Article intelligence

EngineersAdvanced

Key points

SPEAR applies code-as-action to automatic prompt engineering for free-form agentic optimization.
Python sandbox enables structural error analysis like confusion matrices and error clustering.
Auto-rollback and guard metric ensure monotonic improvement.
Outperforms existing methods on both industrial tasks and academic benchmarks.

Why it matters

This matters because SPEAR applies code-as-action to automatic prompt engineering for free-form agentic optimization.

Technical impact

May affect model selection, inference cost, product capability, and evaluation benchmarks.

[2605.26275] SPEAR: Code-Augmented Agentic Prompt Optimization

[Submitted on 25 May 2026]

Title:SPEAR: Code-Augmented Agentic Prompt Optimization

View a PDF of the paper titled SPEAR: Code-Augmented Agentic Prompt Optimization, by Mengyin Lu and 8 other authors

View PDF

Abstract:Automatic prompt engineering (APE) rewrites prompts to improve downstream task performance, but existing APE loops treat the optimizer itself as a fixed pipeline. We port the code-as-action paradigm of CodeAct (Wang et al., 2024a) to APE and propose SPEAR (Sandboxed Prompt Engineer with Active Roll-back), a free-form agentic optimizer with four tools -- evaluate, python, set_prompt, finish -- that decides autonomously how and when to use them. The distinctive tool is the Python sandbox: the optimizer writes and executes arbitrary Python on the current evaluation DataFrame, performing structural error analysis (confusion matrices, error clustering, per group metrics) the agent itself authors. Two guardrails turn the long-horizon agent into a monotone-improving optimizer: auto-rollback on metric regression, and an optional guard metric floor. We evaluate on three industrial LLM-as-judge suites (13 judge tasks across recruiter-intake, conversational-memory, and query-refinement systems) plus seven BBH tasks and GSM8K. SPEAR wins every industrial task on the primary metric ($\kappa$ 0.857 vs 0.359 on tool-selection; F1-macro 0.815 vs 0.763 on filter-relevance; $\kappa$ 0.254 vs 0.218 on the hardest extraction dimension). On BBH-7 SPEAR averages 0.938 accuracy vs GEPA 0.628 and TextGrad 0.484. Ablations show the Python tool is the largest single lever on complex judge tasks ($\Delta \approx +0.79\kappa$ on the 5-class tool-selection judge, $\Delta \approx +0.35\kappa$ on the hardest extraction dimension when removed); its irreplaceable contribution is class-pair confusion aggregation that a long-context LLM cannot extract reliably from the raw eval DataFrame.

Comments: 19 pages, 3 figures, EMNLP 2026 submission

Subjects:

Computation and Language (cs.CL)

Cite as: arXiv:2605.26275 [cs.CL]

(or arXiv:2605.26275v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2605.26275

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Huimin Han [view email] [v1] Mon, 25 May 2026 19:01:10 UTC (327 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled SPEAR: Code-Augmented Agentic Prompt Optimization, by Mengyin Lu and 8 other authors

View PDF

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-05

Change to browse by:

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)