AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。
Most fine-tuning teaches a model to produce a better answer to a single prompt. Agentic tasks are different. An agent that files an expense report or triages a support ticket takes a sequence of actions, and each step depends on what came before. There’s rarely one correct sequence to imitate, and the only reliable feedback is whether the whole sequence succeeded. Supervised fine-tuning has no labels for that, and single-turn reinforcement learning cannot credit a decision made three steps before the outcome. Multi-Turn Reinforcement Learning (MTRL) on Amazon SageMaker AI trains models on exactly that signal. SageMaker AI sends prompts to an agent you host on Amazon Bedrock AgentCore, or on your own infrastructure, such as Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Elastic Compute Cloud (Amazon EC2), or AWS Fargate, through an AWS Lambda forwarder. The example here uses AgentCore. The agent calls the policy model being trained, acts in your environment over as many turns as the task needs, and reports one reward when the episode ends. You don’t specify the right trajectory, only how to score the result. This post uses Wordle, a game in which a player has six tries to guess a hidden five-letter word, as the example. After each guess, every letter turns green if it is in the right position, yellow if it’s in the word but in a different position, and gray if it isn’t in the word. The game suits MTRL because the rules are few, each guess depends on the feedback from the last, and the outcome is scored objectively. Wordle is a teaching example, and frontier models already play it well. In practice, you would use MTRL to make a small, low-cost model good at a multi-step task specific to your business, such as an internal workflow that public models were never trained on. The same pipeline applies to that setting. In our run, MTRL took gpt-oss-20b from a 61 to an 84 percent solve rate on held-out words at its default reasoning effort, and from 15 to 69 percent at the low effort it trained at, for about $339 in training tokens. In this post, we walk through the pipeline end-to-end: deploying the agent and reward to AgentCore, preparing the dataset, training, monitoring in MLflow, and evaluating at serving settings. We also share the reward-design mistakes and training failures we hit. Your datasets, agent, and trained model stay in your AWS account and are not used to train the base models. All code is in the accompanying aws-samples repository. Solution overview The system has two halves. SageMaker AI owns the training job and the policy model. AgentCore hosts the agent, the Wordle environment, and the reward function. The following diagram shows how they connect. Figure 1: MTRL on SageMaker AI, where the training job invokes the AgentCore-hosted agent once per rollout to sample the policy model, play the game, and report a reward For every rollout, the MTRL service invokes the AgentCore runtime, a capability of Amazon Bedrock AgentCore, with one training row, which holds one secret word. A Strands agent in the runtime calls the policy model through the SageMaker Job Runtime’s OpenAI-compatible endpoint. The model plays up to six guesses through one tool, guess_word, and after each guess the environment returns per-letter feedback and a summary of what is known. When the game ends, the agent returns one reward. Group Relative Policy Optimization (GRPO) uses the spread of rewards across several rollouts of the same word to update a Low-Rank Adaptation (LoRA) adapter on the policy model. Three design points matter for what follows: The agent and reward live in one place. A single Python file deployed to AgentCore holds the environment, the tool, and the scoring function. The dataset carries only the secret word. The word lists bundled with the agent are built from the open-source SCOWL word lists and are used only for scoring. The reward is outcome-dominant. A solve pays 1.05–1.55 depending on speed, a loss pays -0.5, and never making a valid guess pays -1.5. A small shaping term rewards good strategy within each tier. Supported models and AWS Regions MTRL is serverless: you pick a base model and the service provisions the training and sampling infrastructure. At the time of writing, it supports the following base models, each trained with a LoRA adapter: Base model AWS Regions Amazon Nova 2 Lite US East (N. Virginia) (us-east-1), US West (Oregon) (us-west-2) GPT-OSS-20B US East (N. Virginia) (us-east-1), US West (Oregon) (us-west-2) Gemma 4 31B (instruction-tuned) US West (Oregon) (us-west-2) Qwen 3.6 27B US West (Oregon) (us-west-2) We chose GPT-OSS-20B because it has the lowest per-token price of the four and its reasoning effort is controllable per request. See the MTRL documentation for the current list. Pricing You don’t choose instances or size a cluster. You pay per million tokens in three dimensions, priced by base model: Prefill: input tokens the policy model processes. These accumulate in a multi-turn task, because every turn re-reads the history. Sample: tokens the policy model generates, including reasoning tokens. Train: tokens processed in the backward pass that updates the LoRA adapter. The 100-step run consumed 491.6 million prefill, 159.7 million sample, and 643.3 million train tokens across 105,486 rollouts in 12 hours 40 minutes. At GPT-OSS-20B list prices in the US East (N. Virginia) Region (us-east-1) at the time of writing, that is about $339: Dimension Tokens Price per 1M Cost Prefill 491,616,806 $0.12 $58.99 Sample 159,737,098 $0.30 $47.92 Train 643,308,554 $0.36 $231.59 Total 1,294,662,458 $338.50 The train dimension dominates and scales with trajectory length, so shorter games and less verbose reasoning cut the bill. The AgentCore runtime bills separately for the vCPU and memory the agent uses. For this run, that was about $79, from the runtime’s Amazon CloudWatch usage metrics at us-east-1 list prices. DescribeJob reports the token counts under BillableTokenUsage. See Amazon SageMaker AI pricing for current rates. Prerequisites To follow this walkthrough, you need the following. See also the MTRL prerequisites. Python 3.10 or later, uv, Node 20 or later, and the AgentCore CLI (npm install -g @aws/agentcore). See Get started with Amazon Bedrock AgentCore for installation and the IAM permissions the CLI needs. botocore 1.43.37 or later, on the machine that launches the job. This is the first version that includes the AgentRFT CreateJob operation. Two AWS Identity and Access Management (IAM) roles, described in Step 1. An Amazon Simple Storage Service (Amazon S3) bucket in the same Region as the agent. The training job must also run in that Region, because CreateJob rejects a cross-Region runtime ARN. Step 1: Create the IAM roles MTRL uses two roles, and one of them has a trust policy that the general SageMaker AI documentation doesn’t mention. Agent runtime role. agentcore deploy creates this role for you. It trusts bedrock-agentcore.amazonaws.com and needs the AmazonSageMakerJobRuntimeAccess managed policy, which lets the agent call the policy model and report rewards. The CDK stack in the sample repository attaches this policy automatically. Training job role. You create this one. It must trust job.sagemaker.amazonaws.com with both sts:AssumeRole and sts:TagSession, and it needs the AmazonSageMakerJobFullAccess managed policy. { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"Service": "job.sagemaker.amazonaws.com"}, "Action": ["sts:AssumeRole", "sts:TagSession"] }] } Note: The trust principal is job.sagemaker.amazonaws.com, not sagemaker.amazonaws.com. With the generic principal, CreateJob fails with “Could not assume role,” and the message names the correct principal only after retries. Step 2: Deploy the agent to Amazon Bedrock AgentCore Clone the sample repository, set your account and Region in Wordle/agentcore/aws-targets.json, and deploy: git clone https://github.com/aws-samples/sample-multi-turn-rl-wordle cd sample-multi-turn-rl-wordle/Wordle agentcore validate agentcore deploy The CLI builds the agent’s dependencies for the runtime’s Linux ARM64 platform, creates the execution role with AmazonSageMakerJobRuntimeAccess attached, and deploys through AWS CloudFormation. We recommend it over uploading a code package on the console, where it’s common to get the roughly 700 MB of compiled dependencies or the policy attachment wrong. Note the runtime ARN in the output for the training job. Figure 2: The Wordle agent runtime in the Amazon Bedrock AgentCore console with status Ready The runtime reads two environment variables from agentcore.json. REASONING_EFFORT=low sets reasoning_effort on every request to the policy model. In our evaluation jobs, low effort used 63 to 83 percent fewer sample tokens per game than medium. REASONING_PROMPT_HINT=true also adds “Reasoning: low” to the system prompt, which is how gpt-oss reads the setting in its Harmony chat format. Neither variable gives the model any hint about the secret word. Figure 3: Runtime environment variables showing REASONING_EFFORT set to low The agent is one file, main.py. Simplified, its entrypoint looks like the following. For your own task, you change the task parsing, the tools, and the reward: app = BedrockAgentCoreApp() @app.entrypoint # outermost: starts the HTTP server AgentCore invokes @sagemaker_rft_handler # reports the returned reward to SageMaker AI def handle_rollout(payload): prompt, answer = parse_task(payload["prompt"]) game = WordleGame(answer) @tool def guess_word(guess: str) -> str: """Submit a 5-letter Wordle guess and get feedback.""" return game.guess(guess) agent = Agent(model=build_model(), system_prompt=SYSTEM_PROMPT, tools=[guess_word]) agent(prompt) return {"reward": game.final_reward()} Important: The sagemaker_rft_handler decorator does not start an HTTP server by itself. Wrap it with BedrockAgentCoreApp and @app.entrypoint, with the entrypoint decorator outermost. Without this, every invocation times out after 30 seconds. Step 3: Verify the runtime Before spending GPU time, confirm the whole chain works with a payload shaped like the one the training service sends. The repository includes one as Wordle/payload.json: { "prompt": "{\"prompt\": \"Guess the 5-letter word\", \"answer\": \"beach\", \"id\": \"smoke-test\"}", "metadata": { "jobArn": "arn:aws:sagemaker:us-east-1:111122223333:job/AgentRFT/smoke-test", "rolloutId": "r1" } } The fake job ARN is expected to fail, and the way it fails is the proof. Put your own account ID into it first: for a job ARN in another account, the service returns a 403 before it checks whether the job exists, which looks like a broken IAM setup. From the Wordle directory, run: ACCOUNT=$(aws sts get-caller-identity --query Account --output text) sed "s/111122223333/$ACCOUNT/" payload.json > /tmp/payload.json aws bedrock-agentcore invoke-agent-runtime \ --agent-runtime-arn \ --runtime-session-id smoke-test-$(date +%s)000000000000000 \ --payload fileb:///tmp/payload.json /dev/stdout In the runtime’s Amazon CloudWatch logs, you see the agent parse the prompt, authenticate to the Job Runtime endpoint, and receive a 400 error reading “Could not find job.” That failure confirms that parsing, authentication, and error reporting work. The command itself prints {“reward”: -1.5}, because the agent never reached the policy model. A 403 reading “not authorized to access this resource” means the job ARN still names another account. A 30-second timeout instead means the HTTP server is not starting (see the Important note in Step 2). Step 4: Prepare the dataset The MTRL dataset has no target responses, transcripts, or labels. Each row is one task for the agent to attempt. Here are the first three rows of the training file: {"prompt": "{\"prompt\": \"Guess the 5-letter word\", \"answer\": \"riser\", \"id\": \"wo [truncated for AI cost control]