待翻譯：How to Debug AI Agents

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

Conceptual Guide Agent observability powers agent evaluation January 27, 2026 16 min Go back to blog Create agents Key Takeaways You can't build reliable agents without understanding how they reason, and you can't validate improvements without systematic evaluation. This article explains the primitives for agent observability, how to evaluate agents at different granularities, and how production traces become the foundation for continuous improvement. TL;DR You don't know what your agents will do until you actually run them — which means agent observability is different and more important than software observability Agents often do complex, open-ended tasks, which means evaluating them is different than evaluating software Because traces document where agent behavior emerges, they power evaluation in a multitude of ways When something goes wrong in traditional software, you know what to do: check the error logs, look at the stack trace, find the line of code that failed. But AI agents have changed what we're debugging. When an agent takes 200 steps over two minutes to complete a task and makes a mistake somewhere along the way, that’s a different type of error. There’s no stack trace - because there’s no code that failed. What failed was the agent’s reasoning. From debugging code to debugging reasoning You still write code to define your agent, e.g. which tools exist, what data is available. You write prompts and tool descriptions to guide the agent's behavior, but you won't know how the LLM will interpret these instructions until you run it. The source of truth thus shifts from code to traces that show what the agent actually did. Agent engineering is an iterative process, and tracing + evaluation are how you close the loop. In this post, we'll explore why agent observability and evaluation are fundamentally different from traditional software, what new primitives and practices you need, and how observability powers evaluation in ways that make them inseparable. Agent observability ≠ software observability Pre-LLMs, software was largely deterministic — given the same input, you'd get the same output. Logic was codified. You could read the code and know exactly how the system behaved. When something went wrong, logs pointed you to which service or function failed, then you'd go back to the code to understand why it happened and fix it. AI agents break the assumptions of determinism and code as a source of truth. As we're moving from traditional software to LLM applications to agents, each step introduces more uncertainty. LLM apps make a single call to an LLM with context, introducing natural language's inherent "fuzziness" but remain constrained to one LLM call. However, agents call LLMs and tools in a loop until they determine a task is done — and can reason across dozens or hundreds of steps, calling tools, maintaining state, and adapting behavior based on context. When building an agent, you attempt to recommend the application logic in code and prompts. But you don't know what this logic will do until actually running the LLM. Traditional software vs. LLM apps vs. Agents When something goes wrong, you're not finding a single line of code that failed. Instead, you're asking: Why did the agent decide to call edit_file instead of read_file at step 23 of 200? What context and prompt instructions informed that decision? Where in this two-minute, 200-step trajectory did the agent go off track? Traditional tracing tools can't answer these questions. A 200-step trace is too large for a human to parse, and traditional traces don't capture the reasoning context behind each decision; they only capture which services were called and how long each took. Agent evaluation ≠ software evaluation Traditional software testing relies on deterministic assertions: write tests that check output == expected_output, verify they pass, then ship. Online evaluation (A/B tests, product analytics) measures business impact separately. Evaluating agents differ from evaluating software in a few key ways: 1. You're testing reasoning, not code paths Traditional software has tests at different levels of granularity (unit, integration, e2e), testing deterministic code paths you can read and modify. Agents also need testing at different levels, but you're no longer testing code paths — you're testing reasoning: Single-step: Did the agent make the right decision at this moment? Full-turn: Did the agent perform well in an end-to-end execution? Multi-turn: Did the agent maintain context across a conversation? 2. Production becomes your primary teacher In traditional software, you can catch most correctness issues with offline tests (unit tests, integration tests, staging). You still test in production through canary deployments and feature flags, but the goal is to catch edge cases and integration issues you missed. With agents, production plays a different role. Because every natural language input is unique, you can't anticipate how users will phrase requests or what edge cases exist. Production traces reveal failure modes you couldn't have predicted and help you understand what "correct behavior" actually looks like for real user interactions. This shifts how you think about evaluation: production isn't just where you catch missed bugs. It's where you discover what to test for offline. Production traces become test cases, and your evaluation suite grows continuously from real-world examples, not just engineered scenarios. The primitives of agent observability Agent observability uses three core primitives to capture non-deterministic reasoning: Runs: A single execution step (one LLM call with its input/output) Traces: A complete agent execution showing all runs and their relationships Threads: Multi-turn conversations grouping multiple traces over time These use the same concepts as traditional observability (e.g. traces, spans) but capture reasoning context rather than service calls and timing Runs: capturing what the LLM did at a single step A run captures a single execution step. This is most useful for capturing how the LLM behaved at a particular point in time. This captures the complete prompt for an LLM call, including all instructions, tools used, and context. These runs serve dual purposes: Example of an agent run and its components For debugging: See exactly what the agent was thinking at any step. What was in the prompt? What tools were available? Why did it choose this action? For evaluation: Write assertions against this run. Did the agent call the right tool? With the right arguments? Traces: capturing trajectories A trace captures a complete agent execution by linking together all the runs that occurred. A reasoning trace captures: All information about what goes into the model at each step, captured as runs that make up the trace All tool calls with their arguments and results The nested structure showing how steps relate to each other Agent traces are massive. While a typical distributed trace might be a few hundred bytes, agent traces can be orders of magnitude larger. For complex, long-running agents, traces can reach hundreds of megabytes. This context is necessary for debugging and evaluating the agent's reasoning. Example trace and its components Threads: multi-turn conversation context A single trace captures one agent execution, but agents often operate across sessions involving multiple interactions with a user or system. A thread groups multiple agent executions (traces) into a single conversational session, preserving: Multi-turn context: All interactions between user and agent in chronological order State evolution: How the agent's memory, files, or other artifacts changed across turns Time span: Conversations can last minutes, hours, or days Example of a thread capturing multiple turns of conversation Consider debugging a coding agent that worked fine for 10 turns but suddenly started making mistakes in turn 11. The turn 11 trace in isolation might show the agent calling a reasonable tool. But when you examine the full thread, you discover that in turn 6 the agent updated its memory with an incorrect assumption, and by turn 11 that bad context had compounded into buggy behavior. Threads are essential for understanding how agent behavior evolves over time and how context accumulates (or degrades) across interactions. How this influences agent evaluation Agent behavior only emerges at runtime, and is only captured by observability (runs, traces, and threads). This means that to evaluate behavior you need to evaluate your observability data. This raises two key questions: At what granularity do you evaluate agents? At the run, trace, or thread level? When do you evaluate agents? If behavior only emerges when you run the agents, can you evaluate them offline in the same way you do software? Evaluations agents at different levels of granularity You can evaluate agents at different levels of granularity, which map 1:1 to the observability primitives. What you're evaluating determines which primitive you need: Single-step evaluation validates individual runs → Did the agent make the right decision at a specific step? Full-turn evaluation validates complete traces → Did the agent execute the full task correctly? Multi-turn evaluation validates threads → Did the agent maintain context across a conversation? Single step vs. full turn vs. multi-turn evaluation patterns Single-step evaluation: unit tests for decisions Sometimes, you need to validate a specific decision point without running the entire agent. You may want to see if the agent choose the right tool in a specific scenario, or whether it used the correct arguments. This is like a unit test for agent reasoning: set up a specific state (conversation history, available tools, current task), run the agent for one step, and assert that it made the right decision. Single-step evaluation validates runs, i.e. individual LLM calls. Example: Testing a calendar agent's tool selection A scheduling agent needs to find available meeting times before scheduling. You want to verify it checks availability first rather than immediately trying to create the meeting: Your single-step test: Setup state: Conversation history = user said "Schedule a meeting with Harrison tomorrow morning", available tools = [find_meeting_times, schedule_meeting, send_email] Run one step: Agent generates next action Assert: Agent chose find_meeting_times (not schedule_meeting) Why you need runs: Single step tests often come from real production cases that error. In order to recreate these, you need the exact state of the agent before that step. Detailed run captures are the only way to get this! Single-step evaluations are efficient and catch regressions at individual decision points. In practice, about half of agent test suites use these single-step tests to isolate and validate specific reasoning behaviors without the overhead of full agent execution. ‍ 2. Full-turn evaluation: end-to-end trajectory assessment Other times, you need to see a complete agent execution. Full-turn evaluation validates traces, i.e. complete agent executions with all their runs, and let you test multiple dimensions: Trajectory: Did the agent call the necessary tools? For a coding agent fixing a bug, you might assert: "The agent should have called read_file, then edit_file, then run_tests." The exact sequence might vary, but certain tools must be called. Final response: Was the output correct and helpful? For open-ended tasks like research or coding, the quality of the final answer often matters more than the specific path taken. State changes: Did the agent create the right artifacts? For a coding agent, you'd inspect the files it wrote and verify they contain the correct code. For an agent with memory, you'd check that it stored the right information. Testing t [truncated for AI cost control]