待翻译：LLM optimization integration for Amazon SageMaker Python SDK

AI 服务暂时不可用，以下为来源正文，待恢复后补全翻译。

Optimizing generative AI inference deployments requires benchmarking endpoints, evaluating instance configurations, and iterating on deployment settings. The Amazon SageMaker Python SDK v3 now exposes generative AI inference recommendations in Amazon SageMaker AI directly in your notebook workflow. These recommendations are also accessible through the Amazon SageMaker AI UI and Boto3 APIs. With this release, you can benchmark an endpoint, generate data-driven deployment recommendations, and deploy the recommended configuration directly from a notebook using the Amazon SageMaker Python SDK v3. In this post, we demonstrate how to use the new SDK interface for the end-to-end workflow to optimize generative AI inference deployments. Benefits of generative AI inference recommendations in Amazon SageMaker AI Generative AI inference recommendations in Amazon SageMaker AI automate inference optimization by: Benchmarking a live Amazon SageMaker endpoint against a synthetic or real-traffic workload, measuring throughput, time-to-first-token (TTFT), end-to-end latency, and more. Generating deployment recommendations ranked by cost-performance tradeoff using your actual usage patterns. Deploying the top-ranked configuration directly to an Amazon SageMaker real-time endpoint. Previously, these capabilities required using Amazon SageMaker Studio or constructing AWS SDK for Python (Boto3) API calls. With this launch, they become Python SDK operations, fitting naturally into existing notebook and pipeline workflows. New SDK interfaces The new functionality is available under the sagemaker.serve.ai_inference_recommender package starting with version 3.17.0 and exposes the following primary operations: Entry point What it does ModelBuilder.from_jumpstart_config(…) Builds a ModelBuilder from a JumpStart model ID and compute config start_benchmark(endpoint, …) Runs a load test against a deployed endpoint with a configurable synthetic workload mb.generate_deployment_recommendations(…) Explores instance/framework configs against your workload and returns ranked recommendations mb.deploy(…) Deploys the top recommendation to a real-time endpoint ModelBuilder.from_recommendation_job(job_name) Hydrates a ModelBuilder from a completed recommendation job — deploy in a different process or session Prerequisites Verify you have the latest version of the Amazon SageMaker Python SDK installed: pip install --upgrade sagemaker >= 3.17.0 You will also need: An AWS account with an AWS Identity and Access Management (IAM) role with Amazon SageMaker execution permissions. A deployed Amazon SageMaker real-time endpoint (or a JumpStart model to deploy; see the following section). Solution overview Consider a common scenario: you have a generative AI model ready for production and need to determine the optimal instance type, framework configuration, and serving parameters. Traditionally, this involves manual trial and error across multiple instance types, container versions, and concurrency settings. With the Amazon SageMaker Python SDK integration, you can automate this entire workflow in a single notebook. The following walkthrough guides you through the end-to-end journey using this notebook: Generate deployment recommendations: Let the service explore instance and framework configurations against your workload profile and return ranked options. Interpret and select: Review the ranked results, understand the tradeoffs, and pick the best fit. Deploy: Push the winning configuration to a live Amazon SageMaker endpoint. Benchmark: Validate the deployed endpoint under realistic load conditions. Compare frameworks: Optionally run LMI and vLLM head-to-head to find the best serving stack. Generate recommendations from real traffic data Your first step is to find the best deployment configuration for your model and workload. Rather than manually deploying across multiple instance types, call mb.generate_deployment_recommendations(…) to let the service explore instance types and framework configurations against your workload profile. The service deploys your model on each candidate, runs a load test matching your traffic pattern, and returns a ranked list of configurations optimized for your chosen performance target. import time, uuid from sagemaker.core.jumpstart.configs import JumpStartConfig from sagemaker.serve import ModelBuilder from sagemaker.train.configs import Compute from sagemaker.serve import InferenceFramework, PerformanceTarget uid = f"{int(time.time())}-{uuid.uuid4().hex[:8]}" src_model_name = f"demo-rec-source-{uid}" rec_ep_name = f"demo-rec-ep-{uid}" mb = ModelBuilder.from_jumpstart_config( jumpstart_config=JumpStartConfig(model_id=MODEL_ID), compute=Compute(instance_type=INSTANCE_TYPE), role_arn=ROLE, ) source_model = mb.build(model_name=src_model_name) rec_job = mb.generate_deployment_recommendations( tokenizer="google/gemma-4-e2b-it", concurrency=1, request_count=10, prompt_input_tokens_mean=32, output_tokens_mean=32, streaming=True, performance_target=PerformanceTarget.TTFT_MS instance_types=[INSTANCE_TYPE], advanced_optimization=False, framework=InferenceFramework.LMI, role_arn=ROLE, wait=True, ) #Comparative table across all returned recommendations print(mb.recommendations) # .best is the top-ranked row top = mb.recommendations.best print(f"Throughput avg: {top.expected_performance.request_throughput.avg}") print(f"TTFT p99: {top.expected_performance.time_to_first_token.p99}") # auto_approve=True bypasses the ModelPackage approval-status check rec_endpoint = mb.deploy( endpoint_name=rec_ep_name, role=ROLE, wait=True, auto_approve=True, ) print(f"Deployed: {rec_endpoint.endpoint_name} ({rec_endpoint.endpoint_status})") Recommendation results can also be represented as a Python data frame. import pandas as pd pd.set_option("display.width", 200) pd.set_option("display.max_columns", None) pd.set_option("display.max_colwidth", 60) rows = [] for i, rec in enumerate(mb.recommendations): raw = rec._raw spec = raw.model_details.inference_specification_name for m in raw.expected_performance: rows.append({ "rank": i, "spec_name": spec, "instance": raw.deployment_configuration.instance_type, "metric": m.metric, # ← attribute, not subscript "stat": m.stat, "value": float(m.value), "unit": m.unit, }) df_long = pd.DataFrame(rows) print(df_long) rank spec_name instance metric stat value unit 0 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge RequestThroughput avg 112.7664 Requests/Second 1 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge OutputTokenThroughput avg 3608.5300 Tokens/Second 2 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge RequestLatency p50 462.1300 Milliseconds 3 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge RequestLatency p90 999.5400 Milliseconds 4 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge RequestLatency p99 1069.8400 Milliseconds 5 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge TimeToFirstToken p50 438.5300 Milliseconds 6 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge TimeToFirstToken p90 983.3300 Milliseconds 7 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge InterTokenLatency p50 0.7900 Milliseconds 8 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge InterTokenLatency p90 2.8700 Milliseconds 9 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge ClientSideConcurrency 64.0000 Count 10 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge RequestThroughput avg 96.8522 Requests/Second 11 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge OutputTokenThroughput avg 3099.2700 Tokens/Second 12 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge RequestLatency p50 541.1600 Milliseconds 13 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge RequestLatency p90 1122.2000 Milliseconds 14 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge RequestLatency p99 1162.5300 Milliseconds 15 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge TimeToFirstToken p50 502.9000 Milliseconds 16 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge TimeToFirstToken p90 1088.4800 Milliseconds 17 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge InterTokenLatency p50 1.0000 Milliseconds 18 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge InterTokenLatency p90 3.6100 Milliseconds 19 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge ClientSideConcurrency 64.0000 Count How to interpret recommendation results The recommendations table shows two candidate configurations (rank 0 and rank 1), both on ml.g6.2xlarge but with different LMI container versions. Here’s how to read the key metrics and choose between them: Key metrics to compare: RequestThroughput (avg): Requests the endpoint can serve per second. Higher is better. OutputTokenThroughput (avg): Total tokens generated per second across all concurrent requests. Higher is better. RequestLatency (p50/p90/p99): End-to-end time from request to full response. Lower is better. TimeToFirstToken (p50/p90): How quickly the user sees the first streamed token. Lower is better. InterTokenLatency (p50/p90): Delay between successive tokens during streaming. Lower is better. Choosing between the two configurations in this example: Rank 0 (lmi-26-0-0) delivers 112.8 req/s throughput and 3,609 tokens/s, with p90 TTFT of 983 ms and p90 latency of 1,000 ms. Rank 1 (lmi-27-0-0) delivers 96.9 req/s throughput and 3,099 tokens/s, with p90 TTFT of 1,088 ms and p90 latency of 1,122 ms. Rank 0 wins on every dimension: approximately 16% higher throughput and approximately 10 percent lower latency. The service ranks it first because the job was configured with performance_target=PerformanceTarget.TTFT_MS, meaning the optimizer prioritized configurations that minimize time-to-first-token. General decision framework Latency-sensitive applications (chatbots, interactive UIs): Prioritize low TTFT (p90/p99) so users perceive fast responses. Throughput-sensitive workloads (batch summarization, offline processing): Prioritize high RequestThroughput and OutputTokenThroughput to maximize tokens per dollar. If two configurations are close on your primary metric, use the secondary metrics as tiebreakers, then factor in cost (a smaller instance at similar performance saves money). In this example, the top-ranked configuration (lmi-26-0-0) is the clear choice because it dominates across all metrics at the same concurrency level (64). Deploy from previously run recommendation job In production workflows, you often generate recommendations in one session and deploy in another. For example, a data scientist might run the recommendation job during experimentation, while an MLOps pipeline deploys the result during a release cycle. Use ModelBuilder.from_recommendation_job(job_name) to hydrate a ModelBuilder from a completed job: from sagemaker.serve import ModelBuilder # Hydrate a fresh ModelBuilder from a completed recommendation job mb = ModelBuilder.from_recommendation_job("my-rec-job-name") print(f"Loaded {len(mb.recommendations)} recommendations") print(mb.recommendations) # Deploy the top-ranked recommendation endpoint = mb.deploy( role=ROLE, wait=True, auto_approve=True, ) Deploy a JumpStart model and benchmark it After you have deployed your recommended configuration, the next step is to validate its performance under controlled conditions. Benchmarking confirms that the endpoint meets your latency and throughput requirements before serving production traffic. The SDK makes this straightforward: deploy a JumpStart model and run a synthetic load test in only a few lines of code. import time, uuid from sagemaker.core.jumpstart.configs import JumpStartConfig from sagemaker.serve import ModelBuilder, start_benchmark from sagemaker.train.configs import Compute from sagemaker.serve import InferenceFramework, PerformanceTarget uid = f"{int(time.time())}-{uuid.uuid4().hex[:8]}" ep_name = f"demo-bench-ep-{uid}" model_name = f"demo-bench-model-{uid}" # Build and deploy a JumpStart endpoint mb = ModelBuilder.from_j [truncated for AI cost control]