待翻译：Agentic Data Operations Platform (ADOP): Data engineering into hours

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

Data engineering teams routinely spend weeks standing up a single new data source: writing ETL, hand-writing quality checks, updating semantic models, and validating compliance. The Agentic Data Operations Platform (ADOP) on AWS is designed to significantly accelerate that timeline. It’s a reference architecture built on Amazon Bedrock and your AI coding tool of choice. Specialized AI agents automate the full Bronze to Silver to Gold lifecycle, with configurable controls designed to support your data governance and regulatory compliance efforts. For Heads of Data Engineering, three things change. Engineers stop spending the majority of their time on pipeline plumbing and start shipping data products. Compliance moves from a downstream gate to an inline control applied at onboarding time. And your architecture, not the model, governs how every AI coding tool (Claude Code, Kiro, Cursor, Codex) interacts with your data systems. This blog post is for VPs of Engineering, Chief Data Officers, and Data Platform Directors, with implementation detail for platform engineers later in the post. Figure 1: Six data engineering challenges that ADOP addresses The agents in dev, artifacts in prod This is the design choice that separates ADOP from a typical agentic platform pitch. ADOP is a build-time accelerator, not a runtime dependency. Agents run in development environments where they reason, propose, and generate: ETL code, quality checks, semantic layer definitions, regulation controls. Engineers review the output. Continuous integration and continuous delivery (CI/CD) promotes the generated artifacts (deterministic PySpark, SQL, Airflow DAGs, IAM and Cedar policies) into staging and production. In ADOP’s default pattern, production runs deterministic artifacts without calling a model. Organizations that require model-in-the-loop inference at runtime can extend this architecture using Amazon Bedrock endpoints, but the generated pipeline code itself remains static and auditable. Figure 2: ADOP token economics and return on investment How ADOP differs from general-purpose coding assistants: Those are general-purpose coding assistants: brilliant, but open-ended. Point them at a data platform and every engineer gets a different architecture on a different day. ADOP is opinionated on purpose. It wraps those same models in: A narrowed lane – data-engineering skills and prompts, not “anything you can type.” Company philosophy baked in – your standards live in the design, not in someone’s memory. No large language model (LLM) freelancing on architecture – the model fills in the blueprint. It doesn’t draw it. Policy and regulation guardrails – apply controls that support your compliance efforts at build time, not only at review. One onboarding flow for the whole enterprise – every source lands the same way, every time. General tools make a developer faster. ADOP makes every developer consistent. How ADOP relates to Amazon Bedrock AgentCore: Amazon Bedrock AgentCore is a platform to build, connect, and optimize agents at scale, with any framework or model. ADOP runs agents in development and ships deterministic artifacts to production. Both are valid AWS aligned patterns. ADOP optimizes for cost predictability and audit posture on regulated data workloads. Use cases ADOP applies wherever data engineering velocity is throttled by manual onboarding and compliance overhead. Common patterns include: Enterprise data onboarding at scale – describe a new source in natural language. Agents handle schema inference, ETL, quality checks, and semantic layer updates. Regulated pipelines in healthcare and financial services – configurable controls designed to help you address regulatory requirements in your industry, applied per dataset through dedicated governance prompts. Customers are responsible for determining their own compliance. AI-ready Gold layers populated and maintained automatically for business intelligence and machine learning (ML) features. Multi-tool AI development governance – Claude Code, Kiro, Cursor, and Codex all operate from the same architectural contract. Architecture ADOP is an AI-powered coding framework that builds end-to-end data pipelines on AWS and multi-cloud environments. It launches a Data Onboarding Agent on Claude Code through Amazon Bedrock, using Claude Code’s Dynamic Workflow feature to spawn specialized sub-agents for each stage of pipeline construction. Figure 3: ADOP architecture overview, with the Data Onboarding Agent spawning specialized sub-agents on Amazon Bedrock Figure 4: ADOP lakehouse layers from Bronze to Silver to Gold, with built-in compliance controls Sub-agents – Sub-agents handle metadata generation, data ontology deduction, data quality checks, ETL transformations, and orchestration (Airflow or AWS Step Functions). Requirements are enriched iteratively through conversational interaction with user persona, and every artifact is validated locally before deployment to AWS with human-in-the-loop approval. Decision engine (AI clone) – The Decision Engine acts as an AI-encoded version of your enterprise architect, embedding your organization’s guidelines, technology standards, and design philosophy directly into the build process. This helps promote consistency across builders, alleviating the fragmentation that occurs when teams use general-purpose coding tools without shared guardrails. Guardrails – Sub-agents are constrained by the architectural contract: tool routing rules, Cedar authorization policies, invariants, and inline compliance prompts. While the reference implementation targets AWS, the framework extends to other services with a CLI or Model Context Protocol (MCP) interface, supporting hybrid and multi-cloud environments. Data compliance – Three capabilities round out the architecture. ADOP helps you apply compliance-related controls: one regulation prompt per governance framework can be applied at onboarding, so legal reviews a prompt file, not application code. You remain responsible for validating that controls meet your regulatory obligations. Agent observability – Every agent decision is traced through AgentTrace (intent, tool selected, outcome, cost) and publishable to Amazon CloudWatch or an OpenTelemetry sink for audit. And the entire stack runs locally in dev by default. When scale demands it, promote to AgentCore runtime, a capability of Amazon Bedrock AgentCore, with no change to the architectural contract. Responsible AI and data handling – Agents might process regulated or personally identifiable data during development. Customers should review their data-handling practices, apply appropriate access controls, and validate that agent behaviors align with their organization’s responsible-AI policies before promoting artifacts to production. How to get started in two steps Start by cloning the repository. git clone https://github.com/aws-samples/sample-Agentic-Ai-Data-Operations.git Upload a dataset to Amazon Simple Storage Service (Amazon S3) or local storage, then run a modified prompt. Note: The following example uses fictitious data, bucket names, and field references for illustration purposes only. No real personally identifiable information (PII) is represented. This example doesn’t constitute regulatory compliance guidance or legal advice. /onboard-workflow Onboard attendance data from s3://amzn-s3-demo-source-bucket/demo_landing/attendance.csv into Silver with dedup on (employee_id, check_in) and not-null policy on employee_id and check_in,and into a flat denormalized Gold Iceberg table aggregated daily-per-employee with derived measures(hours_worked_clean, attendance_rate, late_arrival_flag, overtime_hours, absence_category). Run daily at 03:00 UTC. Apply data governance controls: hash/pseudonymize PII fields in Silver, suppress or mask sensitive fields in Gold, enforce retention policies, and log processing metadata. Apply guidelines (This example is illustrative only and does not constitute compliance guidance.) Please profile the data first, then propose your recommended quality thresholds and transforms before generating any code. Figure 5: Running the ADOP onboarding workflow in Claude Code on Amazon Bedrock ADOP: proof of concept to production The early weeks are architecture-heavy because encoding your standards (not building pipelines) is the one-time investment. After the contract exists, each new source is a prompt, not a project. Directionally, teams running this pattern have seen source onboarding timelines compress significantly on subsequent sources, with the curve flattening further as the skill-trace memory accumulates. Figure 6: A phased ADOP adoption timeline from foundation to production Change management Transitioning to agent-driven data engineering requires deliberate organizational change. The following plan facilitates smooth adoption across engineering teams while preserving accountability and quality standards. Stakeholder communication – Identify three communication tiers: executive sponsors (CDO, VP Engineering) receive monthly progress dashboards. Platform and data engineering leads get weekly sprint summaries. Individual contributors receive real-time updates through team channels. Frame messaging around what ADOP preserves (engineering judgment, architectural standards) rather than what it automates. Publish a one-page FAQ addressing common concerns about agent-generated code quality and job impact before the first enablement session. Training schedule – Week 1: AWS-led ADOP workshop covering architecture contract setup, decision engine configuration, and platform best practices. Week 2: Hands-on prompt authoring lab. Each team onboards one low-risk source end-to-end with AWS guidance. Week 3: Artifact review and guardrail configuration session. Engineers validate agent output against their own code. Weeks 4–6: Office hours twice weekly for troubleshooting. Reduce to weekly from Week 7 onward. Record all sessions for asynchronous onboarding of future team members. Phased rollout strategy – Phase 1 (Weeks 1–3): Pilot with two to three engineering champions and one non-critical data source. Champions validate output quality and provide feedback to refine the architectural contract. Phase 2 (Weeks 4–6): Expand to the full platform team. Onboard 3–5 additional sources of increasing complexity. Phase 3 (Weeks 7–12): Organization-wide rollout. New source onboarding flows through ADOP. Existing pipelines migrate opportunistically during scheduled maintenance windows. Success metrics — Track four key indicators: (1) Source onboarding cycle time, targeting significant reduction by Phase 3. (2) First-pass artifact acceptance rate, with targets defined based on your organization’s quality standards. (3) Engineering satisfaction score through anonymous pulse surveys at Weeks 3, 6, and 12. (4) Guardrail compliance rate, measuring how consistently generated pipelines pass automated policy checks without manual intervention. Escalation paths — Level 1: Engineering champions resolve prompt-authoring questions and minor artifact adjustments within their squad. Level 2: Platform team addresses architectural contract gaps, guardrail misconfigurations, or recurring artifact rejections within one sprint. Level 3: VP of Engineering or CDO intervenes for cross-team adoption blockers, resource conflicts, or policy disputes that cannot be resolved at the platform level. Document all escalations in a shared log to identify systemic issues and feed improvements back into the architectural contract. Security and data privacy A common concern with agent-driven development is how the build process handles secrets, credentials, and sensitive data. ADOP addresses this through several design choices. Secrets management – Secrets don’t enter the agent context. Database credentials, API keys, and service tokens are resolved at deploy time [truncated for AI cost control]