翻訳待ち：Automated Reasoning policy refinement in Amazon Bedrock

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

Refining an Automated Reasoning policy in Amazon Bedrock has been a manual cycle of diagnose, hand-edit, retest, and repeat. Today, we are announcing automatic policy refinement, which automates the diagnose-and-fix work in that cycle. The refinement engine diagnoses failing tests and proposes formal-logic fixes. You approve every change before it takes effect. Automated Reasoning checks in Amazon Bedrock Guardrails use formal verification to prove answer correctness. On unambiguous translations from natural language to formal logic, they deliver up to 99% verification accuracy, as reported in the GA announcement. To get started, you build an Automated Reasoning policy from a source document and validate it with test cases. Customers told us that this iterative tuning creates the biggest friction point in policy development. In this post, we walk through two new refinement modes: Iterative Refinement for rule issues, and Ambiguous Variable Refinement for language issues. For each mode, we show a complete API workflow (start, poll, retrieve) and a repeatable console workflow for turning failing policies into passing ones. What Automated Reasoning checks actually are Automated Reasoning checks translate natural language into formal logic, then apply automated reasoning techniques to produce a finding: VALID, INVALID, SATISFIABLE, IMPOSSIBLE, or TRANSLATION_AMBIGUOUS. For a full introduction to how policies work, refer to our GA announcement post. For this post, the key concept is the two-step validation pipeline. First, the translate step maps natural-language input/output to variable assignments using the variable descriptions in your policy. Second, the validate step applies your formal rules to those assignments. When a test fails, the root cause lives in one of those two steps, and each refinement mode targets a different one. Figure 1 traces that pipeline end to end. Figure 1: How Automated Reasoning checks validate a response at runtime. Automated Reasoning checks translate natural language into variables using the policy’s variable descriptions, then validate those variables against the policy’s formal rules to return a finding. This two-step pipeline is why refinement has two modes. Testing your policy. You validate a policy by attaching tests: each test is input/output text plus the result you expect. Run tests individually or as a batch. Failures tell you exactly where the policy diverges from your intent. Why policies need refinement: Two failure modes Recall the two-step pipeline: translate (natural language to variable assignments) and then validate (formal logic to finding). A failed test means one of these steps produced something you didn’t expect. Automated Reasoning checks surface two distinct failure signals that map cleanly to each step. Failure mode 1: Rule issues (the logic is wrong) In a rule-issue failure, the translation works correctly: the right variables have the right values, but the validation result doesn’t match your expectation. The problem lives in your rules: a rule is too permissive, too restrictive, or missing entirely. Concretely, you expected INVALID but got SATISFIABLE because a missing or too-permissive rule lets a bad answer through. Or you expected SATISFIABLE but got INVALID because an overly strict rule blocks a correct answer. Mental model: The system understood the question perfectly but applied the wrong logic. You need to fix the rules. Failure mode 2: Translation ambiguous (the language is wrong) When a test returns TRANSLATION_AMBIGUOUS, the validation engine runs and produces different outcomes depending on which interpretation it follows. In some cases, the translation models disagreed on how to map the natural-language input to your policy’s variables, and each competing interpretation led to a different validation result. The finding surfaces two or more options, each with its own translation and conclusion, plus differenceScenarios showing where the interpretations diverge in practice. Common root causes include overlapping variable definitions (“tenure” compared to “years of service”), vague descriptions, and inconsistent value formats (5 compared to 0.05 for “5%”). Matching mode to failure This table summarizes which refinement mode addresses which failure type: Failure mode Root cause Refinement mode What it does Rule issues Logic is wrong Iterative Refinement Proposes rule or variable additions, edits, or deletions Translation ambiguous Language is ambiguous Ambiguous Variable Refinement Proposes clearer variable descriptions that collapse multiple interpretations into one Use Ambiguous Variable Refinement when the system cannot determine a single translation. The next two sections walk through each mode in turn: what it does, when to use it, how the review gate works, and how to launch it programmatically. We start with Iterative Refinement because rule-issue failures are the more common case. Iterative Refinement: Fixing the rules When tests fail because the logic is wrong (the translation is clean but the validation result doesn’t match your expectation), the problem lives in your rules. Iterative Refinement (ITERATIVELY_REFINE_POLICY) automates the diagnose-and-fix cycle so you don’t need to manually trace each rule, hypothesize a correction, and hand-edit formal logic. Consider a policy with 10–30 rules. Previously, a fix would take a subject matter expert multiple rounds of manual diagnosis and hand-editing of SMT-LIB formal logic. That work now compresses to a single review-and-approve step, with no formal logic written by hand. How it works Iterative Refinement takes three inputs. The first is your existing policy definition (the current rules, variables, and types). The second is a source document containing the authoritative natural-language text that describes how things should work. The third input is optional: natural language feedback with explicit instructions describing the change you want. For example, the feedback field might contain: “Update the tenure requirement for parental leave from 12 months to 6 months, as specified in section 3 of the revised document.” Given these inputs, the refinement engine analyzes how the current rules diverge from the source document and your feedback. It proposes a set of candidate changes (new rules, edited rules, added variables) that bring the policy in line. The convergence loop Iterative Refinement, as the name suggests, iterates. Behind the scenes, the engine generates a candidate change, simulates its effect on your saved tests, checks whether the previously failing tests now pass, and adjusts if they don’t. This can involve several internal cycles for a single request, especially when a fix in one rule ripples into others. The iteration happens internally, though: you don’t observe each intermediate attempt, and you don’t need to shepherd it. What you receive is the converged result: a proposed diff that shows exactly which rules changed, which variables changed, and how the change affects every test in your suite. The review gate After convergence, the Review policy changes screen appears. You then select Accept changes or Discard changes. Accepting writes the changes to your DRAFT policy. Discarding leaves everything exactly as it was. Prerequisites and when to use Iterative Refinement requires at least one test attached to your policy. Without a failing test signal, there’s nothing to drive the refinement. Use this mode when the translation is correct (right variables, right values) but the validation result is unexpected. Do not use it when the finding is TRANSLATION_AMBIGUOUS. That’s a language problem better addressed by Ambiguous Variable Refinement. Through the API Refinement runs as an asynchronous build workflow. Using the AWS SDK for Python (Boto3), the flow has four steps: export the current policy definition, start the workflow, poll for completion, and retrieve the proposed changes. Set buildWorkflowType to ITERATIVELY_REFINE_POLICY. The iterativeRefinementContent block accepts one to five source documents (required) and up to 4,000 characters of optional feedback: import boto3 bedrock = boto3.client("bedrock") # Export the current policy definition (required input to the workflow). policy_definition = bedrock.export_automated_reasoning_policy_version( policyArn=policy_arn, )["policyDefinition"] with open("hr-leave-policy.pdf", "rb") as f: source_document = f.read() response = bedrock.start_automated_reasoning_policy_build_workflow( policyArn=policy_arn, buildWorkflowType="ITERATIVELY_REFINE_POLICY", sourceContent={ "policyDefinition": policy_definition, "workflowContent": { "iterativeRefinementContent": { "documents": [ { "document": source_document, "documentName": "hr-leave-policy.pdf", "documentContentType": "pdf", } ], "feedback": "Update the tenure requirement from 12 to 6 months per section 3.", } }, }, ) build_workflow_id = response["buildWorkflowId"] The call returns immediately with a buildWorkflowId, not the proposed changes. The workflow moves from SCHEDULED to BUILDING until it reaches COMPLETED, FAILED, or CANCELLED. Convergence typically takes one to a few minutes, depending on policy size. Poll get_automated_reasoning_policy_build_workflow until the status reaches a terminal state. Then retrieve the converged proposal with get_automated_reasoning_policy_build_workflow_result_assets, requesting the POLICY_DEFINITION asset to see the updated rules (and BUILD_LOG for the action log): import time while True: workflow = bedrock.get_automated_reasoning_policy_build_workflow( policyArn=policy_arn, buildWorkflowId=build_workflow_id, ) status = workflow["status"] if status in ("COMPLETED", "FAILED", "CANCELLED"): break time.sleep(10) if status == "COMPLETED": assets = bedrock.get_automated_reasoning_policy_build_workflow_result_assets( policyArn=policy_arn, buildWorkflowId=build_workflow_id, assetType="POLICY_DEFINITION", ) proposed_definition = assets["buildWorkflowAssets"]["policyDefinition"] The returned policy definition is the proposed DRAFT, the full new definition. To commit it, call update_automated_reasoning_policy with this definition. To see what changed, diff it against the policy definition you exported before starting the workflow. The console Review policy changes screen wraps this same start-poll-retrieve sequence, rendering the diff behind the Accept changes and Discard changes buttons. Ambiguous Variable Refinement: Fixing the language Iterative Refinement handles rule issues, but not every failing test is a rule issue. When the translation itself is unstable, no amount of rule-editing will help. You need to fix the language the policy uses to describe its variables. That is what Ambiguous Variable Refinement does. It follows the same asynchronous start-poll-retrieve pattern and lands on the same review-and-accept screen. The difference is in the proposals. They center on variable descriptions and merges, with rule and type updates applied as needed to keep the policy consistent. When tests produce TRANSLATION_AMBIGUOUS results (refer to failure mode 2 earlier in this post), competing translations lead to different validation outcomes. Ambiguity can also come from how the validated content itself is phrased. This section focuses on ambiguity in the policy variables. How it works Translation ambiguity, because of policy variable issues, typically stems from a handful of root causes. Overlapping variables occur when two variables describe the same concept. For example, tenureMonths (“How long the employee has worked in months”) and monthsOfService (“The employee’s months of service”) both capture employment duration. As a result, translation models disagree on which one to use. Incomplete descriptions arise when a variable’s description is too vague to guide translation. Inconsistent value formatting creates ambiguity when the system [truncated for AI cost control]