待翻译：KnowledgeForge: mining gold from the ITSM ticket graveyard

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

KnowledgeForge is about mining gold from the IT Service Management (ITSM) ticket graveyard: the resolved incident tickets whose knowledge never reaches a knowledge base article. Enterprise IT support teams resolve thousands of tickets every month, and each one holds something useful: a symptom, a root cause, and the fix an engineer applied. That knowledge stays locked in ticket history, where the next engineer to hit the same problem won’t find it. The knowledge base itself has the opposite problem. It grows, but it grows messy: duplicate articles pile up, content goes stale, and quality varies depending on who wrote each article and when. A support engineer searching for an answer wades through near-identical drafts, some accurate and some three product versions out of date. We built KnowledgeForge to work both sides of that gap. It mines resolved incident tickets for new articles. At the same time, it curates the existing knowledge base by sorting articles by type, removing duplicates, scoring their quality, and rewriting weak content. A knowledge manager reviews and approves the result, so a person still owns what goes live. This post covers the AWS building blocks behind KnowledgeForge: Amazon Bedrock for generation and content improvement, Amazon S3 Vectors, a capability of Amazon Simple Storage Service (Amazon S3), for duplicate detection, and AWS Step Functions for orchestration. For each one, we explain why we chose it and link to the documentation. If you’re building a large-scale document-processing pipeline on generative AI, you can reuse these patterns. Prerequisites To deploy and follow along with this solution, you need: An AWS account with access to Amazon Bedrock, and access enabled for Anthropic Claude Sonnet 4.5 and Amazon Titan Text Embeddings V2 models. Permissions to create the resources this solution uses: Amazon S3 buckets and Amazon S3 Vectors indexes, AWS Step Functions state machines, AWS Lambda functions, Amazon Elastic Container Service (Amazon ECS) services on AWS Fargate, Amazon DynamoDB tables, Amazon Simple Queue Service (Amazon SQS) queues, Amazon Bedrock guardrails, and AWS Key Management Service (AWS KMS) keys. The AWS Cloud Development Kit (AWS CDK) installed, and working familiarity with Amazon Bedrock, AWS Step Functions, Amazon ECS, and vector embeddings. The code from the aws-samples/sample-knowledgeforge repository. A closed-loop knowledge base lifecycle KnowledgeForge is two subsystems that feed each other. Generation turns clustered incident tickets into new draft articles. When a group of related tickets describes the same problem, the system writes a knowledge base article and a root cause analysis document from that cluster. Curation then takes every article, both newly generated and existing, through four steps: classify it by type, check for duplicates, score its quality, and improve weak content. Finished articles go to ServiceNow for knowledge-manager review. The loop closes because curation stores a vector for every article, and generation reads those vectors back before writing anything new. The following diagram shows how the two subsystems connect. Figure 1: The closed-loop knowledge base lifecycle. Ingestion feeds generation and curation, a knowledge manager reviews the output, and curation embeds every article so generation can reuse those vectors as grounding on the next run. At a high level, an article moves through five stages: Ingestion – Resolved tickets and existing articles land in Amazon S3, tracked by a data catalog. Generation – Clustered tickets become new draft articles on Amazon ECS with AWS Fargate. Curation – Each article is classified, deduplicated, quality-scored, and improved through an AWS Step Functions workflow on AWS Lambda. Human review – Enriched articles go to ServiceNow for knowledge-manager approval, and the decision is written back to Amazon DynamoDB. Closed loop – Curation embeds every article into the Amazon S3 Vectors index, and generation reuses those vectors on the next run. The two subsystems run on different compute, for reasons the next sections explain as they follow an article through the system. Generating articles from incident clusters Generation starts with a cluster of tickets that share a theme. An upstream process groups resolved incidents by the problem they describe and drops the result into an Amazon S3 bucket as a JSON file, scoped to one customer. Each theme carries keywords, an article scope, and a sample of ticket descriptions and work notes. A new file in the bucket sends an event to an Amazon SQS queue. A container on Amazon ECS with AWS Fargate polls the queue, reads the file, and processes up to five themes at once for a customer. Before writing, the system grounds itself in what exists. For each theme, it retrieves the five most similar existing articles from that customer’s Amazon S3 Vectors index and passes them to the model as reference context. This Retrieval Augmented Generation (RAG) keeps terminology consistent and reduces invented procedures. When no reference articles exist yet, the system generates from the ticket data alone and flags the procedures for review. Generation runs on Anthropic Claude Sonnet 4.5 in Amazon Bedrock. For each theme, the model produces two documents with a fixed structure: Knowledge base article: Title and summary, symptoms, root cause, resolution steps, prevention, and related topics. Root cause analysis document: Executive summary, problem description, customer impact, five-why analysis, workaround and resolution, corrective and preventive actions, timeline of key events, and cause code. We use response streaming from Amazon Bedrock so the container assembles each document as tokens arrive rather than waiting for the full response. Why containers instead of functions We run generation on Amazon ECS with AWS Fargate because of the shape of the work. Generating two full documents for a theme can take several minutes, and a busy file holds many themes, so a single unit of work can run for a long time. The workload also arrives in bursts, quiet for stretches and then a large batch at once. A long-running container service that scales its task count on queue depth, and scales back when the queue drains, fits this pattern well. AWS Fargate matches this profile. It runs our containers serverlessly, scales on demand as themes arrive, and lets the team focus on the generation logic rather than on managing compute capacity. Each document lands in Amazon S3 as JSON, ready for curation. Finding duplicate articles with Amazon S3 Vectors The first curation challenge is detecting whether an article already exists. Duplicates come in several forms. Two articles describe the same fix in different words, a newer article supersedes an older one, or an engineer copies an article, changes two lines, and saves it as new. We chose Amazon S3 Vectors to solve this. It stores embedding vectors directly in Amazon S3, which avoids a standalone vector database. It keeps metadata alongside each vector for per-customer filtering and bills per query and per gigabyte rather than per running node. That makes it affordable to keep a vector for every article in the library. If you already store content in Amazon S3, you can add vector search without new infrastructure. For details, see the Amazon S3 User Guide. Keyword matching misses these duplicate forms, so the pipeline matches on meaning. Every article gets a 1,024-dimension embedding from Amazon Titan Text Embeddings V2, stored in a per-customer Amazon S3 Vectors index. A vector store usually powers retrieval. Here the same index doubles as a duplicate detector. A new article is embedded, the index is queried for the nearest existing vectors, and anything inside a tight cosine-distance threshold counts as a duplicate. We start with a cosine distance of 0.05 (a similarity of 0.95 or higher) and a top-K of 5. We tuned the distance by sampling flagged pairs and tightening it until near-identical articles matched without catching merely related ones. A looser threshold produced false duplicates, and a tighter one missed reworded copies. The query is one call, filtered to the current customer and to active articles: response = s3vectors.query_vectors( vectorBucketName=VECTOR_BUCKET, indexName=VECTOR_INDEX, queryVector={"float32": embedding}, topK=TOP_K + 1, # +1 because the article can match itself returnMetadata=True, returnDistance=True, filter={"$and": [ {"tenant_id": {"$eq": TENANT_ID}}, # per-customer isolation {"status": {"$in": ["UNIQUE", "PENDING"]}}, # ignore retired articles ]}, ) matches = [v for v in response.get("vectors", []) if v["key"] != article_id] When the system finds a duplicate pair, it keeps the fresher article and retires the stale one instead of dropping the newcomer by default. The newest accurate version wins, which is the behavior a support engineer wants. Reusing the index this way helps on retries too. Embeddings cost a model call to produce. A re-run reads the stored vector back instead of recomputing it, saving both time and Amazon Bedrock spend when a batch reruns. Orchestrating curation at scale with AWS Step Functions Curation runs over batches of articles that need orchestration to spread work across workers and recover from failures. AWS Step Functions provides that with a two-phase distributed map. It manages workflow state, applies the retries and error handling declared in the definition, and fans work across workers without custom coordination code. For how the distributed map works, see the AWS Step Functions Developer Guide. Two configuration choices are worth calling out. First, we set the item processor to STANDARD rather than EXPRESS. Each article makes long-running Amazon Bedrock calls that exceed the 5-minute EXPRESS limit, and STANDARD keeps a full execution history for debugging. Second, we point the ItemReader at a manifest file in Amazon S3 rather than passing items inline, which keeps the workflow state small. A daily schedule on Amazon EventBridge starts the run. An AWS Lambda function finds customers with new or changed articles, groups the changes into batches, and places each batch on an Amazon SQS first-in-first-out (FIFO) queue. The FIFO queue orders batches per customer across runs, using the customer ID as the message group key, so different customers still run in parallel. Ordering across runs is only part of the story. Within a single execution, Phase 2 can process up to 40 batches concurrently, so FIFO alone doesn’t stop two duplicates in different batches from racing. What actually protects deduplication is that it runs sequentially within each batch inside the worker, while quality scoring and improvement run in parallel. Sequential dedup per batch, parallel quality and improvement, and FIFO for cross-run ordering together keep duplicate detection consistent. A dispatcher function pulls one batch at a time and starts a Step Functions execution. The state machine runs in two phases, each a distributed map so the articles in a batch process concurrently: Classify and embed – Each article is classified by type and given a vector embedding. Deduplicate, score, and improve – The pipeline finds duplicates, scores quality, and improves content that falls below the threshold. The following diagram shows the trigger chain, the two phases, and the fault-tolerance mechanisms that protect a run. Figure 2: Curation runs as a two-phase distributed map. Batches are ordered per customer through an Amazon SQS FIFO queue, article content is passed as an Amazon S3 pointer rather than inline state, and a circuit breaker and dead-letter queue keep a bad batch from stalling the run. Passing pointers, not payloads A Step Functions execution carries state between states, capped at 256 KB. Knowledge base articles, with full HTML bodies, pass that limit quickly. Rather than thread article [truncated for AI cost control]