AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。
Observability & Evals LangSmith The Most Common Questions About Agent Evals, Answered October 15, 2026 min Go back to blog Create agents Create agents Key Takeaways Evals score your agent's runs so you can check that it behaves the way you expect. To get started, pick one behavior, build a small dataset from real traces, and run a baseline. Use offline evals before you ship and online evals in production, and turn new failures into dataset examples. Use code evaluators where you can, LLM judges where you can't, and human review to check both. For agents that take many steps, evaluate full threads, trajectories, and final state. Benchmarks like Terminal-Bench are offline evals with longer tasks and their own environments. Harbor runs them and logs the results to LangSmith experiments. As you build and ship an agent, you’ll need a way to check that it behaves the way you expect, both in development and once real users start relying on it. In traditional software, the same inputs produce the same outputs, so a unit test can check for an exact result. Agents are harder to test for a few reasons: Outputs vary between runs, even for the same request More than one answer can be correct, so checking for an exact match falls short The agent chooses its own steps, so a correct output can still come from a “wrong” path. For example, an agent might call a tool it doesn't need or take an action without user permission. Agent evals are the solution. They score your agent’s runs with checks that account for these variations. Similar to the agents they evaluate, these tests are not one-size-fits-all, and the best suite of evals depends on the team, their product, and what they want to optimize for. Over the past few years of working on LangSmith, LangChain’s platform for agent engineering, our team has encountered a lot of different eval questions. This blog walks through some of the most common questions on how to get started with agent evals and the different types of evals you can use in your workflows. Where should we start? Every eval starts with understanding what your agent actually does. A trace is a record of one run of your agent, including its inputs, outputs, and every model and tool call in between. Read through a sample of its traces to see how it handles requests and where it goes wrong. What you find tells you what to test first. Start small. Pick a behavior you care about. For example, for a travel assistant, a behavior could be whether it books the date and price range the user asked for. Build a small dataset from traces. A dataset is a collection of examples, and each example is a single test case with an input and often an expected output. Ten to twenty examples of a behavior is enough to start, and in LangSmith you can add traces to a dataset straight from the tracing view. If your agent isn't in production yet, have your team or a small group of trusted design partners test it out (dogfooding) and work from those traces. Run a baseline. Run your current agent against the dataset and save the result as an experiment. An experiment contains the results of running one version of your agent on a dataset, along with the scores from your evaluators. You’ll compare the performance of future versions of your agent with this experiment. Add new failures as you find them. Each failure you see in production or while testing becomes a new example. The price is accurate, but the agent searched the wrong day. This trace would become a new example in our dataset. Continue adding failures to your dataset and you have an improvement loop. Find a failure, add it to your dataset, improve the agent, and rerun. As you identify more agent behaviors to evaluate, you'll expand your eval suite following the same pattern as above. What's the difference between offline and online evals? Evals can and should run before you ship a change and after your agent is live. Before shipping, evals can tell you whether a change made the agent better or worse. After shipping, evals can tell you how the agent is performing with real users. Offline evals run your agent against a dataset before you ship a change. You choose the example inputs, and you can also include expected outputs (also called reference outputs). Using your dataset, you can compare experiment results across agent versions. They’re best used to test a new prompt, model, or tool, and to catch regressions before they reach users. Benchmarks like Terminal-Bench are also offline evals, built on a fixed set of tasks so you can compare agents, models, or versions. Harbor runs them and logs the results to LangSmith experiments, which we cover in more detail below. Online evals score real production traces as they come in. There's usually no reference output to compare against, so they rely on code checks and LLM judges that can grade a run on its own. They’re best used to monitor quality and find the problems your dataset missed. You need both, and they feed into each other. Failures that online evals flag become new examples in your offline dataset. An example of an offline and online eval for the travel agent. There are a few ways to build a dataset for offline evals in LangSmith. You can write examples from scratch, add runs from your existing traces, send over runs your team reviewed in an annotation queue, upload a CSV or JSONL file, or generate synthetic examples from a few you already have. As your eval suite grows, you'll need to decide whether to add examples to an existing dataset or start a new one. As a rule of thumb: Keep examples in the same dataset when they test the same behavior with the same evaluators, so experiments stay comparable over time. Create a new dataset when the inputs look different (like single requests vs full conversations), when you need different evaluators, or when the dataset serves a different purpose, like a small set that runs on every pull request vs a larger one you run before a release. Within a single dataset, you can also use splits to run a subset of examples. What are the different types of agent evaluators? An evaluator is the logic that scores a run. There are four common types. Most teams start with one or two and add more over time. Code evaluators check things deterministically, like whether the booked price is under the user's budget. They're fast, cheap, and consistent, though their need for deterministic inputs and outputs limit their usefulness. LLM-as-a-judge uses a model to grade things code can't, like whether the assistant's reply clearly confirms the flight details. Human review has your team or subject matter experts score runs directly. It's slower, but it's the best signal for subjective quality and for checking your other evaluators. It’s often used on a smaller sample to then inform an LLM evaluator’s judgement. Pairwise evaluators compare two outputs side by side and pick the better one, which is useful when you're choosing between two prompts or models. Questions you might ask depending on the choice of evaluator Any of these evaluators can be pointed at different parts of an agent’s run. An eval might score: A single step. Ex. Whether search_flights was called with the right date. The final response. Ex. Whether the reply confirms the flight the agent actually booked. A full conversation. Ex. Whether the final booking reflects every change the user asked for across several turns The end state of an environment. Ex. The booking records in a mock booking system after the agent runs in its own container. Benchmarks often use this setup, and you’ll sometimes hear these called containerized evals. Can you trust an LLM judge? An LLM judge uses a model to grade outputs that are difficult to check with code, like whether an answer is supported by the retrieved documents. Judges are flexible, but they can be inconsistent and drift from your team's judgement. A few habits make them reliable: Compare the judge with human scores. Have someone on your team grade a sample of runs and check where the judge disagrees. Adjust the judge's prompt or examples until they line up. Annotation queues in LangSmith make this easy. Our LLM-as-a-judge docs walk you through this. Ask narrow questions. In our travel agent example, asking "Does the reply mention every layover?" is easier to judge than "Is this a good reply?" Define a precise scoring scale. A binary pass or fail, or a small set of labeled categories, is far more reproducible than scoring on a 1-10 scale. A judge may have a hard time distinguishing between a score of 6 vs 7. Contrasting an evaluator with 1-10 scoring (non-descriptive and susceptible to high variability) and a set of binary evaluators. Are evals the same as benchmarks? A benchmark is a type of offline eval composed of a set of tasks. A task is a single test case in a benchmark, including instructions, an environment in a container, and a test script that scores the result. As agents become more capable, a simple LLM judge or code-based eval often isn't enough. Agents can now execute code, edit files, and interface with services. To model this multi-step behavior, benchmarks set up isolated infrastructure in containers. This lets developers test agent behavior offline the same way the agent will work in production. Benchmarks differ from simpler LLM judges and code-based evals in a few ways: Tasks involve more steps. To complete a single task, an agent might install packages, edit files, and run commands across dozens of tool calls. Each task comes with an environment. Agents can now take actions in environments like a sandbox with a code repo, files, and tools. Every task ships with its own isolated environment for the agent to work in. Scoring checks what the agent did. A test script inspects the state of the environment after the run, like whether the tests pass or the right file exists. You can still review the trace, but the score comes from the actions the agent took and the state it left behind. For our travel assistant, a benchmark task might give the agent a mock booking system and a request such as "A storm canceled the traveler's flight to Denver. Rebook them and move their hotel to match." Once the agent finishes, the test script would check the booking records in the database. Each benchmark task comes with its own environment. The test script checks the state the agent leaves behind, and Harbor logs the results across tasks to a LangSmith experiment. Harbor is an open source eval harness that provides the scaffolding for testing on these longer tasks. It's the framework that powers Terminal-Bench, and you can use the same task format to build a benchmark for your own agent. With the LangSmith integration, each Harbor run is logged as a LangSmith experiment. Harbor builds a container for each task, and you can use LangSmith traces to make each environment closer to what your agent sees in production. Benchmark results then sit alongside the rest of your evals, and you can open any run to see the full trace next to its score. We used this setup to improve Deep Agents on Terminal-Bench 2.0 by reviewing traces to find failure modes and then changing the harness. We also used it to build IssueBench, a benchmark for LangSmith Engine. Each IssueBench task gives Engine a batch of traces and scores whether it groups the correct ones into the right issue clusters. Building a benchmark for your own agent follows the same steps as the evals above, with a few changes: Collect tasks from traces. Find failures in LangSmith traces, then turn each one into a Harbor task with its own environment, like a mock booking system populated with a user’s reservations. Define what “good” looks like. Create LLM judges or code-based evals that score based on your rubric. Hill climb on the score. Run the agent and use the scores to A/B test changes. You can set up an agent loop that runs [truncated for AI cost control]