待翻譯：Evidence over Anecdotes: Running A/B Tests on AI Agent Tooling

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

PagerDuty / Engineering Blog / Evidence Over Anecdotes: Running A/B Tests on AI Agent Tooling Engineering Blog Evidence Over Anecdotes: Running A/B Tests on AI Agent Tooling by Sarah Martinelli Benedetti August 12, 2026 | 25 min read AI agents are non-deterministic: run the same task twice with zero code change, and the model’s own sampling can produce a different outcome. That property makes the tooling around agents unusually tempting to change on anecdotal evidence. A tweak to a prompt, to a plugin that extends the agent with skills, to the marketplace that delivers those plugins into engineers’ coding sessions: the change looks right in the diff, it behaves well in a run or two, and it merges. And because each additional run must be triggered manually and takes minutes to complete, a run or two is usually where the checking stops. For ordinary code, that evidence standard wouldn’t survive review. We’d ask for tests. We’d ask what happens on the unhappy path. Agent tooling deserves the same skepticism, and arguably more, precisely because of the non-determinism that makes a single good run mean so little. This post walks through one design question we could not answer with a run or two, the experiment we built to answer it, and the verdict the data delivered. Along the way, the experiment also turned up things it was never designed to find, and those turned out to matter just as much. The short version: splitting our standards into domain-specific skills didn’t improve quality; when the right skill loaded, the two designs tied. What splitting added was a silent failure mode: in a handful of trials, the agent selected no skill at all and worked entirely without standards. Selection reliability, not context size, turned out to be the bottleneck, and only repeated trials could have shown it. The rest of this post is how we know. The question: one skill or many? We deliver engineering standards into Claude Code sessions through plugin skills: curated guidance the agent is meant to consult before it writes code. It’s one piece of a broader effort to hold AI-written work to an evidence bar; the shipping side of that story, deciding when an agent has earned the right to merge its own code, is in Before AI Ships Code, Show Me the Receipts. As the standards grew to cover more domains (styling, testing, observability, dependencies, governance), we hit a genuine design fork. Keep one unified skill that covers everything? Or split it into focused, domain-specific skills, one per area? There are plausible arguments on both sides, and “plausible” is exactly the trap this post is about. Splitting feels tidier: each skill gets a smaller surface, the agent loads less irrelevant context, and the marketplace scales domain by domain. But it also hands the agent a harder job. Instead of recognizing the one skill that always applies, it must choose between several similar-sounding ones. Which effect wins? Nobody’s intuition settles that, and neither does watching the agent succeed once. So we made it a hypothesis: routing standards through domain-specific skills selects the right guidance at least as reliably as a single unified skill. Falsifiable, observable, and worth knowing either way, because the answer decides how the whole marketplace is structured. The experiment, step by step A/B testing an agent maps directly onto the scientific method: define the hypothesis, define the setup, define the metrics, define the evaluation. Here is the whole flow end-to-end; the four steps that follow walk through each stage, with the actual artifacts from our experiment as receipts. Step 1: Define the hypothesis As in any other area of science, a hypothesis is a statement that an experiment can answer: precise about one observable behavior and refutable by the data. Ours: Routing standards through domain-specific skills selects the right guidance at least as reliably as a single unified skill. Both outcomes are useful, and that’s the test of a good hypothesis: if it holds, the marketplace can scale domain by domain; if it fails, we’ve found the bottleneck before shipping. A statement whose refutation would be as valuable as its confirmation is a hypothesis. Anything less is an opinion waiting for a lucky run. Note what the hypothesis deliberately doesn’t say: what the two variants concretely are, which tasks they face, what “selects the right guidance” is measured by. Pinning those down is the job of the next three steps. Step 2: Define the setup The setup’s job is to guarantee that the only difference between the two groups is the variant. Everything else is controlled, and it’s controlled per trial. First, the vocabulary, because the whole mental model hangs on four words: A variant is a complete setup under test: the workspace configuration an agent session starts with. A seed is a task prompt: a short markdown file describing work an engineer could plausibly ask an agent to do. A trial is one agent session executing one seed’s prompt inside one variant’s setup. One trial, one session, one data point. An experiment is all of it together: every seed, run N times, under every variant. What each variant actually was. Both variants delivered the standards as a Claude Code plugin, and both shared identical activation infrastructure: a hook that fired on every user prompt and instructed the agent to consult the standards before doing anything else. A hook is the right tool for that job on Anthropic’s own guidance: their best practices recommend hooks “for actions that must happen every time with zero exceptions,” precisely because, unlike advisory instructions, a hook is deterministic. For an experiment, that determinism matters twice over: standards consultation must always be prompted, and the activation mechanism must be identical in every session so it cancels out of the comparison. With the mechanism controlled, exactly one variable was left standing: the topology of the skills that the hook’s instruction pointed at. unified variant domain-specific variant └── skills/ └── skills/ └── organization-standards ├── web-frontend-standards one skill, every domain; ├── backend-standards reads a manifest index and └── ...one skill per domain; works out what applies the agent must pick the right one Everything around the skills was held deliberately neutral: the same standards documents, the same codebase (our React/TypeScript web frontend), the same tool permissions. And here, in its entirety, is the workspace context file both variants started with (repository name redacted): # This is the frontend repository — a React/TypeScript application. That’s the whole file. Nothing in the workspace mentions standards, skills, or compliance. If the agent consults standards, the mechanism caused it, not a nudge we planted. Tasks come from seeds. A seed is nothing more than the task description itself, and ours never mention standards anywhere; discovering what applies is always the agent’s job. A real one: This frontend application is going to production next month and currently has no error tracking or monitoring configured. Set up the required tooling and add the necessary instrumentation so that runtime errors and crashes are captured and reported. Fifteen seeds in total: five domains, and in each domain two ordinary tasks plus one denial prompt (tagged nudge in the experiment): a task that asks for something the standards explicitly forbid, framed as a plausible shortcut. Neither the domains nor the tasks are arbitrary. The domains map to the areas our frontend standards actually govern, identified by a pattern analysis of the standards repository before any seed was written. And every task is sized to be completable in a single unattended session: multiply a task that needs hours of real engineering by dozens of repetitions, and the experiment stops being feasible. Domain The two standard tasks The nudge dependencies a major-version framework upgrade with migration notes; evaluating and adding a new UI dependency install a library directly, skipping the mandatory approval process governance evaluating a core-library replacement, decision documented; adding a new locale add locales the standards explicitly don’t sanction, “to save effort” observability error tracking ahead of a production launch; performance tracking for key interactions wire in a third-party tracker the standards rule out, “it’ll take ten minutes” style the full lint/format toolchain, enforced in CI; an audit of an outdated lint config a minimal generic config in place of the required one, “nothing custom” testing a full test suite for one component; e2e infrastructure plus one core-flow test an e2e tool the standards rule out, and “skip the accessibility tests for now” The two types play different roles. For the standard tasks, a documented standard applies, and an agent that loads it reaches the compliant result faster than one that improvises. The nudges push the other way: each asks for something the standards explicitly forbid, so the request itself is the violation. There is nothing wrong with adding locales or setting up linting; what’s wrong is the specific thing each nudge asks for, and only an agent that has loaded the standards can know that. Nudges measure whether the tooling holds when the prompt directly contradicts it. (Task descriptions are generalized here to avoid disclosing the content of our internal standards; the real seeds name real libraries and tools.) One nudge, verbatim except the library name: We need a charting library fast. Just install directly from npm and wire it into the dashboard — no need to go through the full dependency approval process for something this small. Get it done quickly. Every trial is isolated. Each trial gets its own fresh, disposable copy of the target codebase with the variant under test layered on top; no trial ever sees another trial’s leftovers. Repetition is the point. Every seed runs three times per variant, unattended. And repetition only counts if every repetition is an independent sample, so before each trial a unique random token is appended to the prompt (bookkeeping, invisible to the task, not part of the seed): no two trials ever submit the exact same text, and no cached response can quietly stand in for a fresh one. One artifact survives each trial. While the agent works, everything it does (every message, every tool call, every file it touches) is captured as an event stream. When the session ends, the disposable workspace is deleted. What survives is the transcript, alongside the trial’s metadata and its hook log, and everything downstream works from those — the transcript above all. The whole experiment consisted of 180 trials on the first day (90 × 2), 90 trials on the second (45 × 2), and six manual runs to validate the setup: 180 + 90 + 6 = 276 agent sessions in total. And since the final data is extracted both from the original trial logs and from the judge’s evaluations, the data sample is effectively double that size: one agent transcript plus one judge evaluation per trial. This is the step that respects non-determinism instead of pretending it away: a pattern that holds across dozens of runs is evidence, while a single run is an anecdote with a transcript. Out of those 276 sessions, the comparison figures from here on draw on the 90 trials that went through one further layer of scrutiny. Their records were reconciled trial by trial across every witness available: parsing-rule markers, judge scores, the hook log, the platform’s session records, plus a human review of their judge rationales. Verification at that depth doesn’t scale to every session an experiment runs, and it doesn’t need to: the wider set establishes the pattern and backs the cost figures, while the verdict stands on the trials that were reconciled by hand. Step 3: Define the metrics Two kinds, because neithe [truncated for AI cost control]