翻訳待ち：Build observable enterprise agentic retrieval using Managed Amazon Bedrock Knowledge Base with AWS CloudFormation

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

Teams that add Retrieval Augmented Generation (RAG) to a foundation model usually start with a single retrieval step against a single knowledge base. That works until the questions get harder, when the answer spans several sources, or the system has to decide which source to consult before it can respond. Enterprise agentic retrieval solves that: an agent reasons about the question, routes it to the right knowledge base, retrieves iteratively, and returns a cited answer. But it introduces a harder operational problem. Once an agent reasons and retrieves in a loop, you can no longer see what it did or whether the answer was any good. A previous post, Build an end-to-end RAG solution using Amazon Bedrock Knowledge Bases and AWS CloudFormation, automated a single-shot RAG workflow with a self-managed (vector-store) Knowledge Base. Amazon Bedrock Knowledge Bases has evolved from RAG to agentic retrieval with the launch of managed knowledge bases. Managed Knowledge Bases agentic retrieval performs multi-turn planning, executes retrieval tools, and generates grounded answers with citations. This post takes the next step: an enterprise agentic retrieval solution where an agent reasons, retrieves across multiple knowledge bases, and synthesizes a cited answer. It is built on the Amazon Bedrock Managed Knowledge Base and Amazon Bedrock AgentCore, with observability and evaluation built in from the start. You deploy all of it with a single AWS CloudFormation chain. Figure 1: End-to-end architecture from synthetic corpora in Amazon S3 through two Managed Knowledge Bases, the AgentCore Gateway (a capability of Amazon Bedrock AgentCore), and the runtime agent to seven layers of observability and evaluation, all deployed by AWS CloudFormation The numbered steps in the architecture diagram map to the workflow for the solution, which is as follows: A user sends a question to the agent hosted on the Amazon Bedrock AgentCore runtime. The runtime auto-instruments every step with OpenTelemetry spans, so the reason-and-act loop is observable from the first call. The agent’s reasoning model plans the task and performs cross-knowledge-base routing: given one retrieval tool per knowledge base, it selects the tool whose topic matches the question (financial or weather). The selected tool call is brokered by the Amazon Bedrock AgentCore Gateway over the Model Context Protocol (MCP), which invokes that knowledge base’s AgenticRetrieveStream API on the Managed Knowledge Base. AgenticRetrieveStream does the within-knowledge-base work: it decomposes the question into sub-queries, retrieves iteratively from the managed datastore (ingested from the corpora in Amazon Simple Storage Service (Amazon S3)), and synthesizes a grounded, cited answer that streams back through the Gateway to the agent. The agent checks whether the returned context is sufficient. If not, it retrieves again in another loop iteration before composing its final answer. If so, it returns the cited answer to the user. Throughout, the runtime emits spans, token usage, and metrics to Amazon CloudWatch and AWS X-Ray, populating the seven observability layers and feeding both the on-demand and continuous evaluation scores. I. Background Before the walkthrough, it helps to establish three things: what makes RAG agentic, why the Managed Knowledge Base is the right foundation for it, and why observability and evaluation belong in the design rather than bolted on later. From RAG to enterprise agentic retrieval Classic RAG does one retrieval and one generation. Enterprise agentic retrieval puts a reasoning agent in the loop: it decides whether and what to retrieve, can retrieve several times to refine, chooses which knowledge base is relevant (semantic routing), and only then composes a grounded answer with citations. This is exactly what the Amazon Bedrock Managed Knowledge Base now delivers as a first-class capability through its AgenticRetrieveStream API. Retrieval is no longer a single lookup but an agent-driven, multi-step process. That produces better answers on complex questions. But it also produces a more complex system to operate, which is why observability and evaluation are built in from the start in this post. Managed compared to do-it-yourself Knowledge Bases Amazon Bedrock now offers a Managed Knowledge Base (Type: MANAGED): Amazon Bedrock manages the ingestion, storage, indexing, and retrieval for you, including embedding and reranking with service-managed models by default, so there is no vector database to provision, scale, or patch. The following table shows the difference between Amazon Bedrock managed and customer-managed knowledge bases: Capability Bedrock Managed Customer-managed (DIY) Agentic retrieval (AgenticRetrieveStream) Supported Not supported AgentCore Gateway integration Supported Not supported Data store Auto scaling, fully managed by Amazon Bedrock You provision, scale, and maintain it Embedding + reranking Built-in managed models (you can select other models available on Amazon Bedrock) You configure them Infrastructure to manage None Vector DB + more Why observability and evaluation An agentic system that “returns an answer” is not enough for production. You need to see how it behaves (latency, call volume, token usage), how good the retrieval and answers are, and you need those signals continuously. This solution ships two CloudWatch dashboards spanning seven layers of telemetry, plus two forms of evaluation (on-demand and continuous), all provisioned by the same templates. II. Solution overview The solution deploys as four native AWS CloudFormation stacks, each wiring its outputs into the next. 01-knowledge-bases creates an Amazon Simple Storage Service (Amazon S3) bucket, two Managed Knowledge Bases (a financial and a weather corpus, so the agent has something to route between), their data sources and IAM, and an ingestion custom resource that uploads the documents and runs the first sync. 02-agentic-gateway stands up an Amazon Bedrock AgentCore Gateway (AWS_IAM auth, MCP) with a per-knowledge-base target built on the native bedrock-knowledge-bases connector, so each knowledge base exposes its own AgenticRetrieveStream tool with no AWS Lambda function or extra container. 03-agent-runtime provisions an Amazon Elastic Container Registry (Amazon ECR) repository and an AWS CodeBuild project that builds an OpenTelemetry-instrumented Strands agent image, the Amazon Bedrock AgentCore runtime that hosts it, the log and trace delivery wiring, and the online evaluation configuration. 04-dashboards creates the two Amazon CloudWatch dashboards. Routing happens at two levels, and it is worth separating them. The agent’s reasoning model does the cross-knowledge-base routing. Given one retrieval tool per knowledge base and a system prompt to pick the tool matching the question’s topic, it decides which knowledge base to consult. The AgenticRetrieveStream API then does the within-knowledge-base work, decomposing the question into sub-queries, retrieving iteratively, and synthesizing a cited answer. So the agent runs a reason-and-act loop. It makes a large language model (LLM) call, decides which knowledge base tool to call, reads what came back through the Gateway, and often retrieves again before composing its final, cited answer. Every step is auto-instrumented by the runtime, so the seven observability layers fill from real traffic. The seven layers each answer a different operational question, and together they cover the agent end to end. Layers 1, 4, and 5 are emitted automatically. Layers 3, 6, and 7 are published as custom metrics by the driver notebook. Layer What it captures Operational question it answers L1 — KB-native metrics Retrieve invocations, errors, throttles per KB Is each knowledge base healthy and serving traffic? L2 — Ingestion Ingestion job status and per-document results Did my documents make it into the knowledge base? L3 — Agentic retrieval quality Reference-free utilization, grounded coverage, duplicate rate Is the agent retrieving relevant, well-grounded context? L4 — Gateway / MCP metrics Gateway tool-call volume and latency Is the retrieval tool layer fast and reliable? L5 — OTEL span tree The agent’s full reason-and-act span trace What did the agent actually do, step by step? L6 — Token usage gen_ai.usage tokens per session and model What is each query costing in tokens? L7 — Evaluation scores Correctness, faithfulness, tool-selection, response relevance Are the answers actually good? Why these services Each choice in this solution follows from the goal of enterprise agentic retrieval that you can operate. The Managed Knowledge Base is the foundation because agentic retrieval and the AgentCore Gateway connector are available only on it. It also removes the vector database you would otherwise provision, scale, and patch. The AgentCore Gateway exposes each knowledge base’s AgenticRetrieveStream as an MCP tool, so the agent gets one tool per knowledge base with no Lambda or extra container to maintain. The AgentCore runtime hosts the agent and auto-emits OpenTelemetry spans, which is what makes Layers 5 through 7 possible without extra wiring. CloudFormation ties it together into one reproducible chain, so the whole system, including the dashboards and continuous evaluation, comes up the same way every time. III. Describe the dataset The solution ships with two small synthetic corpora, bundled in the repository under data/: Financial: A synthetic Octank Financial 10-K (octank_financial_10K.pdf, ~198 KB). Weather: A real, publicly available U.S. Congressional Research Service report on tornadoes (IF12695, tornadoes_report.pdf, ~560 KB). The two corpora are intentionally distinct, so the agent must route each question to the right knowledge base, which is the semantic-routing story. We use two separate knowledge bases rather than one knowledge base with two data sources on purpose. Each knowledge base is exposed as its own retrieval tool, so the agent makes a real routing decision between them. Every per-knowledge-base signal on the dashboards (index size, retrieval quality, token usage, and evaluation scores, all keyed by KnowledgeBaseId) stays cleanly separable. A single knowledge base with two data sources would give the agent only one tool, with no routing to demonstrate and the per-corpus signals merged. Because these are Managed Knowledge Bases, we do not configure chunking, embedding, or an index. On ingestion, Amazon Bedrock parses each PDF, chunks it, embeds it with its managed model, and indexes it automatically. Figure 2: Managed ingestion scans, chunks, embeds, and indexes each document automatically, with one document indexed per knowledge base and zero failures IV. Deploy the solution Deploying the solution takes one command, but it helps to know what that command needs and what it produces. This section covers the prerequisites, the single deploy script, and how to confirm every stack came up. Prerequisites An AWS account with permissions for Amazon Bedrock, Amazon Bedrock AgentCore runtime and Amazon Bedrock AgentCore Gateway, AWS Identity and Access Management (IAM), Amazon CloudWatch, AWS X-Ray, Amazon ECR, AWS CodeBuild, Amazon Simple Storage Service (Amazon S3), AWS Lambda, and AWS CloudFormation. You also need Amazon Bedrock model access enabled for the agent’s model. The agent’s model available in the account (default us.anthropic.claude-haiku-4-5-20251001-v1:0). See Supported foundation models in Amazon Bedrock. CloudWatch Transaction Search enabled, so OpenTelemetry spans land in aws/spans for Layers 5–7. See CloudWatch Transaction Search. AWS Command Line Interface (AWS CLI) v2. Python 3.13 with boto3>=1.43 for the notebook. No local Docker (the agent image is built by CodeBuild). See Install the AWS CLI and Boto3 documentation. One command When the prerequisite steps are comp [truncated for AI cost control]