待翻译：How We Build Agent Environments & Tasks

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

Observability & Evals How We Build Agent Environments & Tasks August 25, 2026 min Go back to blog Create agents TLDR: This is a practical guide on how we create synthetic agent environments and tasks. By the end, we have a two-step pipeline. The first step takes traces, code, and/or human input to build a detailed spec. The second step takes that spec and creates an eval task and an environment for that task. In order to do this, we create a “world spec” to capture shared knowledge, scripts, and important definitions. This process is very iterative. We packaged our process in an updated eval-engineering skill so every team can own this flow for themselves. Glossary: Agent: system that uses an LLM to decide the control flow of an application Environment: where the agent runs Rubric: a set of criteria to judge the output of an agent run Task: An input, an environment, and a test script. Agents run these instructions inside and environment to produce an output which gets scored by a test script Dataset: A collection of tasks, also referred to as a benchmark Evaluation: running an agent over a dataset to get a score Harbor: a framework for defining tasks & datasets and running evals. Traces: detailed run information on an agent’s input and trajectory Task Spec: detailed natural language description about a task (input, environment, test script) that can be used to create a task World Spec: project specific information, scripts, helper functions, or other artifacts that can be used to either help create specs or transform specs into tasks Creating a benchmark To reliably improve agents, you need a good benchmark to run your agents over. This helps identify regression, find areas for improvement, and generally make sure you can iterate on your agent backed by real metrics. Building a good benchmark is hard. Each task in the benchmark needs to have a representative input, an environment that closely aligns to the real world, and an aligned rubric to grade the result on. It takes a lot of time, effort and human alignment to create a single task, let alone a whole dataset. We think a lot about how to create better evals, and being able to more reliably and efficiently create representative benchmarks is a key part of that. Over the past few months we’ve been iterating on a process to help with this. What is the ideal end state? The end state we are working towards is a dataset of high-quality, vetted tasks. These tasks can then be used to evaluate, hill-climb, or post-train our agent. Examples of tasks we create internally at LangChain include: Tasks for GTM engineering testing capabilities like company research & finding individuals despite missing data and entries across customer tables Tasks for prompt optimization where an agent is evaluated Tasks for code review given using data from real merged PRs Tasks for trace mining where a small number of issues exist in a massive corpus of trace data A pipeline to create tasks In order to generate a large number of these tasks, we find it is most efficient to build a pipeline to produce these tasks. We find that a core piece of creating this pipeline is the concept of a “spec”. A spec is a markdown file that describes what the task (input, environment, graders) looks like in natural language. Depending on the dataset you are trying to build, it may have different sections or information required. The benefit of having this concept of a spec is that it separates: figuring out what each task should look like building a task The first step - “figuring out what the task should look like” - can be done in multiple ways, and often requires human iteration to align on what matters to the team and users. The second step - “building the task” - can ideally be more automated with an agent. This separation allows you to create many of different specs, centralize the human review process there, and then parallelize building them. Specs are a good medium for human-agent collaboration and editing. It’s much easier for humans review a markdown spec rather than the raw code and data of a task. Edits can be versioned and tracked just like code and shared across teams. Overall, the ideal end state is a pipeline that consists of two steps: A spec generation step A Spec2Task creation step Assembling world knowledge To do either of those steps well requires gathering specific information about the general dataset you’re trying to create. We call this information “world knowledge” **and it lives in “world spec.” This information is NOT specific to a single task - if it was, it would live in the task spec. Rather, it is general knowledge across all potential tasks in a dataset (ie. the “world”). This knowledge can be in different formats (eg markdown vs python scripts) and used in different ways. For example: During world spec creation Guidance on what pieces of information are helpful to store such as size/shape of data and Scripts for parsing traces (or other data) to extract information During spec to task creation Knowledge of how to create good rubrics for this task (ex: programmatic vs LLM-as-a-judge) Scripts for generating specific data to populate the environment Here are some examples of world knowledge from our internal benchmarks: Prompt optimization benchmark: Which information is important to gather ahead of time to put in the spec (domain, input shape, output classes) Processes and scripts for generating good datasets Standard scoring function to use for all use cases GTM agent benchmark: APIs and schemas for specific backend services (Salesforce, Notion) that need to mock Common questions that users are asking mined from existing agent traces Why generating a world spec is an iterative process In order to generate specs or transform specs into tasks you need a “world spec”. How do you get this world spec and make it useful? We’ve found that the best way to get this spec is work hand in hand with a coding agent to generate a first task, and then have it write up a general world spec that it learned along the way to use in the future. In fact, you may even want to do this for the first two or three tasks to make sure the world spec is truly complete. In order to help with this, we created a skill (eval-engineering) that can help you both create your first task as well as generalize that process into a world spec. What is the coding agent doing in practice when using the eval-engineering skill to create a first task and then build a world spec? Scanning the repository with subagents to fine the exact prompts, tools, skills, etc that an agent interacts with. Grouping traces to find real world patterns of what users are asking the agent to do. These groups are good for brainstorming potential types of tasks Mapping out what credentials would be needed to run an agent. Does the agent call any live tools via APIs such as web search? Should we simulate this behavior or call it live during a Task? Cataloging all services an agent interacts with and their data schemas. Systems like SalesForce or Gong including the tables + schemas the agent interacts with. Finding relationships/hierarchies in data, and planning how good approaches to do synthetic data generation depending on the types of input data. A lot of this knowledge is specific to the agent or domain a user is trying to make Tasks for. A core part creating a world spec is iteratively gathering user feedback, which is why creating this world spec while also generating a first task is so useful. What does a task spec contain The process of generating a single task requires writing down all of the implementation details specific to that task but informed by the overall world knowledge. The spec we create generally should cover three parts: What the agent environment looks like What the inputs should be How the outputs should be scored Some of these may be optional, if they are the same for all tasks and can be covered by the “world spec”. For example: For general QA chatbots, the environment may always be the same (it’s just the inputs/outputs that change). In this case, the environment information could be consolidated in the “world spec” and shared across tasks. For a GTM research task, the inputs may stay fixed (and be specified in the “world spec”) but the environment and therefore the rubric to grade would be part of the task spec Spec2Task Spec-to-task is the pipeline for taking a spec and generating a task in Harbor format. This is most easily done with a coding agent. You can pass it a task specs, give it the “world spec” as a skill containing your specific knowledge on task creation, and the eval-engineering skill (contains some guidance on how to create tasks in Harbor format). Some learnings we found on good Spec2Task creation: Have the pipeline refine tasks by running them with real agents and reading trajectories. This helps them find any flaws in environment design such as overly specific instructions or leaky abstractions. Ex: a poorly written table entry saying “Answer placeholder” The difficultly of tasks can be calibrated by having the pipeline run each task with different tier models (ex: gpt-5.6- Luna vs gpt-5.6-Sol). This gives feedback about whether tasks are too easy or too hard for a certain tier of model or if the task is broken for a stronger model because of a reward hack. Agents are bad at knowing what method to use for generating different types of data. So we give them overall guidance such as using LLMs with rubrics for free-text data and using scripts with sqlite + specified schemas for tabular data. End to end process Use eval-engineering to create a first task Make sure the eval-engineering skill is loaded Example Prompt: “Use the eval-engineering skill, the traces from {LangSmith project}, and the {current repository} to help me create an eval Task for {agent}.” This will involve some back and forth, will create (and surface to user) a separate skill for the world spec, and after agreement will create a Task Review the world spec skill and make any adjustments necessary Example Prompt: Review the {world spec - insert name that was created}skill, tell me the core parts of what it says so I can review it Use “world spec” and eval-engineering to create a second task Switch thread so you can properly validate the world spec skill. Make sure the world spec skill AND eval engineering skill are loaded Example Prompt: “Use the {world spec} skill to create a new task. This task should…” This will involve some back and forth, and will update the {world spec} skill Repeat step 2 Example Prompt: “The customers in this task look too similar. Expand the customer set with bigger and smaller customers with varying amounts of revenue, total employees, emails sent, etc.” Repeat steps 3-4 until confident Scale this process with a coding agent with “world spec” to look at a bunch of traces and generate specs Example Prompt: “Use {world spec} to create 10 new, different task specs for me to review. Use traces from the last 10 days to find new patterns we’re not capturing today in our tasks.” Run each of those specs through a coding agent with “world spec” to create a bunch of tasks Example Prompt: “**Use {world spec} with {task spec X} to create a new task.” Where human judgment is still needed The eval-engineering skill provides a general framework for building agent environments, but the process is not fully autonomous. Agents still need human guidance in two areas. First, refining specs often requires several rounds of feedback. The agent needs help determining whether a spec accurately reflects the real-world domain, user behavior, and task requirements. Second, agents tend to create tasks that are too easy. This helps validate that the environment works, but a useful benchmark needs tasks across a range of dif [truncated for AI cost control]