待翻譯：Evaluate any agent framework with Amazon Bedrock AgentCore Evaluations

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

AI teams building production agents face a frustrating asymmetry: the diversity of agent frameworks keeps growing, but evaluation tooling has not kept pace. Most evaluation systems assume you built your agent in a specific way: a specific SDK, a specific large language model (LLM) client, a specific tracing pattern. The moment you step outside that narrow compatibility zone, the evaluation pipeline breaks. Teams build on LangGraph for its workflow orchestration model, on LlamaIndex for its tight integration with retrieval pipelines, and on the OpenAI Agents SDK when their organization standardizes on GPT models. They use Google ADK for multi-agent coordination, or the Claude Agent SDK for native Anthropic capability. They reach for Strands Agents because its model-driven loop gets a working agent running on Amazon Bedrock AgentCore in minutes rather than days. And increasingly, they deploy all of these on Amazon Bedrock AgentCore runtime, a capability of Amazon Bedrock AgentCore. It handles the hosting, scaling, memory, and observability infrastructure they would otherwise rebuild for each project. Amazon Bedrock AgentCore evaluations solves this fragmentation by decoupling evaluation from the framework choice. Every major framework supports OpenTelemetry, either natively or through a community instrumentation library. As long as an agent’s telemetry flows through OpenTelemetry, the evaluation service can score it, regardless of what SDK sits underneath. This post explains how that works: what telemetry the service reads, how it decides how to read your spans, which attributes carry the evaluation data, and how coverage extends to frameworks beyond the named list. OpenTelemetry as the common language OpenTelemetry is a vendor-neutral instrumentation framework that standardizes how distributed systems emit traces, metrics, and logs. A trace is a tree of spans, and each span represents a single step within a request: one unit of work with its name, timestamps, a set of typed attributes, and optional span events. Spans are exported over the OpenTelemetry Protocol (OTLP) and collected by a telemetry backend. On AgentCore runtime, that backend is the AWS Distro for OpenTelemetry (ADOT), which routes spans and event records to Amazon CloudWatch. An agent’s execution produces many kinds of span, because an agent does many kinds of work. A single user turn can generate spans for model calls, tool calls, document retrieval from a vector store or database, reranking of retrieved results, embedding generation, guardrail checks, prompt-template rendering, memory reads and writes, and the orchestration steps that tie them together. The two conventions name these explicitly: the OpenTelemetry GenAI conventions define operations such as chat, embeddings, retrieval, execute_tool, invoke_agent, create_agent, plan, and a family of memory operations. OpenInference defines span kinds such as LLM, TOOL, RETRIEVER, RERANKER, EMBEDDING, AGENT, CHAIN, GUARDRAIL, EVALUATOR, and PROMPT. A production trace routinely mixes several of these. Out of that full set, the evaluation service needs three span roles to reconstruct what happened in a session and score it: An invoke agent span represents the top-level request-response cycle, which is one user turn in a conversation. It carries the user prompt and the final agent response. Inference spans represent individual model calls, each carrying the message history passed to the model and the model’s reply. Execute tool spans represent each tool the agent called, carrying the tool name, input parameters, and result. These three are what the evaluators operate on. The service classifies every span it receives, reads the values it needs from these three roles, and passes over the rest. Richer traces that also carry retrieval, reranking, guardrail, or memory spans are handled without any special configuration. Those spans simply add context the evaluators don’t require. As frameworks and the conventions add new span kinds over time, this stays forward-compatible: an unfamiliar span kind is context the service skips, not an error. Session (one runtimeSessionId) └── Trace (one user turn, one trace_id) ├── invoke agent span ← read: user prompt + final agent response ├── inference span ← read: messages to model + model reply ├── execute tool span ← read: tool name + parameters + result ├── retriever span (context; not required by evaluators) ├── inference span ← read: next model call with tool result in history └── ... (guardrail, memory, reranker, orchestration spans, and more) The service reads the three marked span roles and treats the others as additional context. Different frameworks and instrumentation libraries record these three roles using different attribute names, nesting structures, and span-naming conventions. The OpenTelemetry GenAI semantic conventions and the OpenInference specification both define schemas for recording these three span roles, but they use different attribute keys and different span-kind vocabularies. AgentCore Evaluations bridges both schemas to the same result. How AgentCore Evaluations reads your traces When you run an evaluation, on demand or through an online evaluation config, the service fetches your agent’s spans and event records from CloudWatch and reconstructs the session. A session is grouped by session.id. Within it, each trace (one trace_id) is one user turn. Every turn is made up of the three span types described earlier. The service classifies each span, extracts the values it needs, and hands the reconstructed session to the evaluators. From that point on, evaluation is entirely framework-agnostic: the same evaluators, GoalSuccessRate, Correctness, Helpfulness, and custom LLM-as-a-judge, score every framework identically. This is shown visually in the following figure. Figure 1: Data flow from AgentCore Runtime through Amazon CloudWatch to AgentCore Evaluations You don’t configure any of this. Every OpenTelemetry instrumentation library stamps a scope.name on the spans and event records it produces, and the evaluation service uses that value to decide how to read them. The correct handling activates automatically from the instrumentation package you installed, with no changes to your agent code. Each supported framework and its scope name is listed in the Supported agent frameworks documentation, which today covers Strands Agents, LangGraph, the OpenAI Agents SDK, LlamaIndex, Google ADK, and the Claude Agent SDK, most with both OpenTelemetry and OpenInference instrumentation. Coverage reaches beyond the named frameworks. Any library whose scope name falls under opentelemetry.instrumentation.* (following the OpenTelemetry GenAI semantic conventions) or openinference.instrumentation.* (following the OpenInference specification) is read through a generic path. In practice, supporting a new framework is usually only a matter of installing a compliant instrumentation package: the scope-name prefix is how a library opts in. A scope named mycompany.agent.tracing will not be picked up, even if its spans follow the conventions exactly. The prefix is the signal that the instrumentation author deliberately conformed to a documented schema. Getting a session evaluated end to end comes down to two requirements. The first is grouping: your agent’s spans must carry a session.id attribute that matches the runtimeSessionId you invoked the agent with. That attribute is what lets the service assemble spans into traces and traces into a session. On AgentCore runtime, ADOT injects this attribute automatically, so no agent code changes are needed. The second requirement is that your data source includes the message content, not only the spans. For agents with unified observability (the default for newly created agents), this is automatic: message content lives in the same per-agent log group as the spans, so a single log group is sufficient. For existing agents still running the pre-unified configuration, however, spans land in the shared aws/spans log group. Message content is stored separately as correlated event records in the agent’s log group. In that configuration, if your data source covers only aws/spans, span classification still succeeds, but message content comes back empty, and any evaluator that scores response quality returns an error. OpenTelemetry GenAI semantic conventions: Attributes that drive evaluation The OpenTelemetry GenAI semantic conventions define a schema for spans produced by LLM frameworks. AgentCore Evaluations reads a specific subset of these attributes to classify spans and extract evaluation data. Knowing which attributes carry that data is useful when you are debugging an evaluation or writing custom instrumentation. Span classification starts with gen_ai.operation.name: invoke_agent marks the invoke agent span, the top-level span per turn, carrying the user prompt and final response. chat marks an inference span. The service reads the message history and the model’s reply. execute_tool marks an execute tool span. The service reads the tool name, input parameters, and result. When gen_ai.operation.name is absent, common in some LlamaIndex traces, the service falls back to traceloop.span.kind, where workflow maps to the invoke agent span, tool to an execute tool span, and llm to an inference span. Tool identity comes from gen_ai.tool.name, and gen_ai.tool.call.id is the correlation ID that links a tool call requested on an inference span to its result on the corresponding execute tool span. The gen_ai.tool.definitions attribute on inference spans carries a JSON-encoded array of tool schemas, which the GoalSuccessRate evaluator uses to verify that the tools the agent called were among those it declared available. Message content lives in one of two places, depending on how telemetry was collected. When telemetry is split, the content is in the correlated event record body (body.input.messages and body.output.messages). When it is not split, the content stays on the span, as gen_ai.input.messages or gen_ai.output.messages attributes or as inline span events. The service reads from whichever location holds the content, so you don’t need to know which path a given deployment took. For the full picture of where content lives, see Spans, event records, and telemetry signals. OpenInference conventions: An alternative semantic layer OpenInference is an open specification maintained by Arize AI and widely adopted across the LlamaIndex, Haystack, and Phoenix ecosystems. It represents LLM operations with different attribute names from the OpenTelemetry GenAI conventions. Several frameworks, including the OpenAI Agents SDK with openinference-instrumentation-openai-agents, Google ADK, and the Claude Agent SDK, produce OpenInference spans. Span classification in OpenInference uses openinference.span.kind. The values LLM, TOOL, AGENT, and CHAIN correspond to inference, execute tool, invoke agent, and structural container spans respectively. Inference-span message content uses a flat, indexed attribute convention. Input messages follow the pattern llm.input_messages.{i}.message.role and llm.input_messages.{i}.message.content, where i starts at zero and increments for each message in the history. Tool result messages carry an additional llm.input_messages.{i}.message.tool_call_id to link them to the corresponding tool call. Output messages follow the same indexed pattern, with tool calls nested further: llm.output_messages.0.message.role = "assistant" llm.output_messages.0.message.tool_calls.0.tool_call.function.name = "get_pto_balance" llm.output_messages.0.message.tool_calls.0.tool_call.function.arguments = "{\"employee_id\":\"EMP-001\"}" llm.output_messages.0.message.tool_calls.0.tool_call.id = "call_abc123" Tool schemas on inference spans use llm.tools.{i}.tool.json_schema, where each value is a JSON string encoding either a plain function sch [truncated for AI cost control]