翻訳待ち：AI-powered metadata correction and harmonization

AI サービスが一時的に利用できないため、復旧後に翻訳を補完します。

As data collection and data generation accelerate, the gap between our ability to produce raw data and our capacity to standardize it continues to widen. Without automation, this gap becomes a critical bottleneck that delays analysis, complicates interpretation, and limits the global value of shared datasets. Metadata harmonization (standardizing labels, identifiers, and formats so datasets from different sources can work together) remains largely manual. AI-powered metadata correction and harmonization offers a way forward, transforming metadata management from a time-consuming responsibility into a process that scales with your data volume and supports open science rather than obstructing it. In this post, we demonstrate how AI-powered metadata correction works in practice, explore two implementation approaches (from human-in-the-loop validation to fully autonomous agent-driven workflows), and provide governance considerations for deploying these solutions in your organization. Metadata correction and harmonization workflow To address this challenge, we developed a centralized metadata correction and harmonization workflow built on AWS, designed to support consistency, interoperability, and accuracy across disparate metadata sources. The system uses Amazon Bedrock for large language model (LLM)-powered schema alignment and correction recommendations, Amazon Simple Storage Service (Amazon S3) for schema and result storage, Amazon DynamoDB for job tracking, Amazon Cognito for authentication, and Amazon Elastic Container Service (Amazon ECS) for compute. The workflow includes a harmonization package that aligns metadata schemas, validates data integrity, and generates correction recommendations. The metadata correction and harmonization system operates as a cyclical workflow that guides data through validation, generates recommendations, and returns control to the user for final approval. The following diagram illustrates this high-level flow: Figure 1: Metadata correction and harmonization workflow As shown in the diagram, the process begins when users upload their metadata files. The system then performs two parallel validation streams: schema alignment verifies that column structures match expected formats, while metadata field validation examines individual field values for compliance. When issues are detected, the system generates targeted correction recommendations and presents them to the user, who maintains final decision authority over changes. This human-in-the-loop approach lets automation accelerate the process while preserving researcher control and domain expertise. Schema alignment The first step in metadata correction is comparing source and target data schemas: verifying that the right columns exist and are properly aligned. Common issues include inconsistent naming conventions (synonyms, misspellings, abbreviations), missing or extra columns, and cases where columns need to be split or combined. Fuzzy string matching can handle basic naming discrepancies, but more complex alignment challenges require a different approach. LLMs available on Amazon Bedrock bring semantic understanding to this problem. Rather than relying on string similarity alone, an LLM-driven approach can recognize industry-specific synonyms, infer meaning from surrounding columns, and detect when a source column should be split into multiple target columns or vice versa. This semantic matching handles the cases that rule-based systems miss. Metadata field validation The metadata field validation component checks whether individual field values comply with schema requirements. The system evaluates each field against predefined rules and categorizes failures into three types. Required field validation identifies mandatory fields that are missing, empty, or whitespace-only. A row missing a required sample identifier, for example, is flagged before further processing occurs. Enumerated value validation compares field contents against controlled vocabularies defined in the schema. If a field accepts only specific instrument types (such as sequencing systems), values outside that list trigger a validation error. This helps prevent free-form text from introducing inconsistencies in standardized fields. Pattern validation applies regular expression matching to verify formatting conventions. Date fields might require YYYY-MM-DD format, while identifier fields might need specific alphanumeric patterns. Values that deviate from the expected format are flagged. Each validation failure is classified and documented with enough context for the recommendation system to generate appropriate corrections. The structured error reports specify the location, type, and nature of each issue. Metadata field recommendation When validation errors are detected during metadata field validation, the system generates recommendations using a layered combination of natural language processing (NLP) and AI-based techniques designed to balance cost efficiency, performance, and interpretability. By prioritizing classical NLP and embedding-based similarity before invoking LLMs, the workflow achieves accurate, scalable recommendations while keeping inference costs predictable. These techniques can operate individually or in sequence within bagging or boosting architectures, dynamically chosen based on confidence thresholds. This adaptive layering lets simpler methods handle routine corrections efficiently, while advanced models resolve only the ambiguous or novel cases that require contextual reasoning. 1. Semantic similarity through embeddings Vector embeddings let the system compare metadata values semantically, identifying close matches based on similarity thresholds. This supports precise correction of common variations such as synonyms, abbreviations, or acronyms. For example, the system can map “Human” to “Homo sapiens” or “NYC” to “New York City.” This approach relies on smaller, efficient embedding models (available on Amazon Bedrock) whose outputs can be cached, providing a cost-effective alternative to LLM-driven reasoning while maintaining accuracy. We evaluated multiple embedding models, including domain-specific biomedical models and Amazon Titan, to identify the most appropriate fit for this workload. We selected Amazon Titan for its strong performance on general and biomedical metadata tasks, commercial availability, and compatibility with Amazon Bedrock model inference. This makes it well suited for organizations that require a supported, scalable embedding solution without the operational overhead of self-managed domain-specific models. 2. Contextual inference Contextual inference lets the system recommend corrections by identifying relationships and shared patterns within the metadata itself. Many datasets exhibit internal consistency where similar rows share structural traits or repeated values across columns. By analyzing these local similarities, the model can infer missing or inconsistent metadata values using only the information present in the current upload, avoiding the need for large external training datasets. In this implementation, contextual inference operates through a hybrid approach that combines distance-weighted k-nearest neighbors with Term Frequency-Inverse Document Frequency (TF-IDF) feature representations and co-occurrence statistics. Each row in the dataset is converted into a composite vector built from its text, categorical, and numeric fields. Text fields are transformed into TF-IDF representations, categorical values are encoded through one-hot or compact learned representations, and numeric fields are scaled to comparable ranges. The system then measures the similarity between rows using cosine or Euclidean distance and predicts the missing value by performing a weighted vote among the most similar examples, where closer neighbors contribute more influence. Further refinement is achieved through co-occurrence analysis using Pointwise Mutual Information (PMI), which identifies value combinations that naturally appear together within the dataset. PMI measures statistical association beyond raw frequency, capturing patterns like “when column A contains X, column B typically contains Y.” The final recommendation combines both similarity and co-occurrence signals with configurable weights. This produces stable and explainable recommendations grounded in internal data structure rather than external training, resulting in efficiency and reliability even in small or specialized datasets. 3. Fuzzy matching Fuzzy search algorithms such as Levenshtein distance detect and correct typographical or formatting inconsistencies. They are especially effective for resolving errors in manually entered metadata, such as misspellings, spacing, or punctuation differences. Fuzzy matching is typically applied early in the correction workflow to capture low-level inconsistencies before semantic or contextual inference layers are applied. 4. LLM-based resolution (fallback layer) When earlier validation methods, including rule-based NLP and embedding similarity models, fail to reach a sufficient confidence level, LLM-based resolution through Amazon Bedrock acts as a fallback layer. These models can reason across complex metadata structures to interpret ambiguous or previously unseen patterns. LLM-based inference is invoked selectively, meaning computationally intensive reasoning is used only when simpler algorithms cannot resolve a field with high confidence. The results are then surfaced for human review to maintain accuracy and traceability in downstream data harmonization. Summary The metadata correction and harmonization workflow combines three stages: schema alignment through LLM-powered semantic analysis, field validation using a tiered rule-based approach, and recommendations driven by embeddings, fuzzy matching, and contextual inference. Simpler methods handle routine corrections, while LLM-based resolution serves as a fallback for edge cases that traditional methods cannot resolve. Prerequisites Before deploying the application, confirm the following tools are installed: Node.js 18+ for AWS Cloud Development Kit (AWS CDK) and frontend development. Python 3.11+ for the API and processor. AWS Command Line Interface (AWS CLI) configured with credentials that have permissions for Amazon ECS (AWS Fargate), Amazon DynamoDB, Amazon S3, Amazon Cognito, Amazon Virtual Private Cloud (Amazon VPC), AWS Identity and Access Management (IAM), AWS CloudFormation, Amazon Elastic Container Registry (Amazon ECR), and Amazon CloudWatch AWS CDK Toolkit (npm install -g aws-cdk) Docker for building container images. Make for running the provided Makefile commands. uv Python package manager for dependency management. node -v # 18 or higher python --version # 3.11 or higher aws --version # AWS CLI configured with credentials Getting started Step 1: Clone and install dependencies Clone the repository and install all Python and TypeScript dependencies: git clone https://github.com/aws-samples/sample-intelligent-metadata-harmonization.git cd metadata-harmonization # Create and activate Python virtual environment make createPythonEnvironment source .venv/bin/activate # Install all dependencies (Python packages, TypeScript packages, CDK, frontend) make install The make install command installs the processor, API, agent, and evaluation packages as editable Python packages using uv, and runs npm install for the infrastructure and frontend projects. Step 2: Configure the deployment Copy the configuration template and edit it with your AWS account details: cp config.yaml.example config.yaml Edit config.yaml: appName: "metadata-harmonization-app" env: "dev" dev: profile: "your-aws-profile" deploymentName: "metadata-harmonization-dev" accountNumber: "123456789012" # Your 12-digit AWS account number region: "us-east-1" deploymentStage: "dev" re [truncated for AI cost control]