LLM optimization integration for Amazon SageMaker Python SDK

Optimizing generative AI inference deployments requires benchmarking endpoints, evaluating instance configurations, and iterating on deployment settings. The Amazon SageMaker Python SDK v3 now exposes generative AI inference recommendations in Amazon SageMaker AI directly in your notebook workflow. These recommendations are also accessible through the Amazon SageMaker AI UI and Boto3 APIs. With this release, you can benchmark an endpoint, generate data-driven deployment recommendations, and deploy the recommended configuration directly from a notebook using the Amazon SageMaker Python SDK v3.

In this post, we demonstrate how to use the new SDK interface for the end-to-end workflow to optimize generative AI inference deployments.

Benefits of generative AI inference recommendations in Amazon SageMaker AI

Generative AI inference recommendations in Amazon SageMaker AI automate inference optimization by:

Benchmarking a live Amazon SageMaker endpoint against a synthetic or real-traffic workload, measuring throughput, time-to-first-token (TTFT), end-to-end latency, and more.

Generating deployment recommendations ranked by cost-performance tradeoff using your actual usage patterns.

Deploying the top-ranked configuration directly to an Amazon SageMaker real-time endpoint.

Previously, these capabilities required using Amazon SageMaker Studio or constructing AWS SDK for Python (Boto3) API calls. With this launch, they become Python SDK operations, fitting naturally into existing notebook and pipeline workflows.

New SDK interfaces

The new functionality is available under the sagemaker.serve.ai_inference_recommender package starting with version 3.17.0 and exposes the following primary operations:

Entry point What it does

ModelBuilder.from_jumpstart_config(…) Builds a ModelBuilder from a JumpStart model ID and compute config

start_benchmark(endpoint, …) Runs a load test against a deployed endpoint with a configurable synthetic workload

mb.generate_deployment_recommendations(…) Explores instance/framework configs against your workload and returns ranked recommendations

mb.deploy(…) Deploys the top recommendation to a real-time endpoint

ModelBuilder.from_recommendation_job(job_name) Hydrates a ModelBuilder from a completed recommendation job — deploy in a different process or session

Prerequisites

Verify you have the latest version of the Amazon SageMaker Python SDK installed:

pip install --upgrade sagemaker >= 3.17.0

You will also need:

An AWS account with an AWS Identity and Access Management (IAM) role with Amazon SageMaker execution permissions.

A deployed Amazon SageMaker real-time endpoint (or a JumpStart model to deploy; see the following section).

Solution overview

Consider a common scenario: you have a generative AI model ready for production and need to determine the optimal instance type, framework configuration, and serving parameters. Traditionally, this involves manual trial and error across multiple instance types, container versions, and concurrency settings. With the Amazon SageMaker Python SDK integration, you can automate this entire workflow in a single notebook. The following walkthrough guides you through the end-to-end journey using this notebook:

Generate deployment recommendations: Let the service explore instance and framework configurations against your workload profile and return ranked options.

Interpret and select: Review the ranked results, understand the tradeoffs, and pick the best fit.

Deploy: Push the winning configuration to a live Amazon SageMaker endpoint.

Benchmark: Validate the deployed endpoint under realistic load conditions.

Compare frameworks: Optionally run LMI and vLLM head-to-head to find the best serving stack.

Generate recommendations from real traffic data

Your first step is to find the best deployment configuration for your model and workload. Rather than manually deploying across multiple instance types, call mb.generate_deployment_recommendations(…) to let the service explore instance types and framework configurations against your workload profile. The service deploys your model on each candidate, runs a load test matching your traffic pattern, and returns a ranked list of configurations optimized for your chosen performance target.

import time, uuid from sagemaker.core.jumpstart.configs import JumpStartConfig from sagemaker.serve import ModelBuilder from sagemaker.train.configs import Compute from sagemaker.serve import InferenceFramework, PerformanceTarget

uid = f"{int(time.time())}-{uuid.uuid4().hex[:8]}" src_model_name = f"demo-rec-source-{uid}" rec_ep_name = f"demo-rec-ep-{uid}"

mb = ModelBuilder.from_jumpstart_config( jumpstart_config=JumpStartConfig(model_id=MODEL_ID), compute=Compute(instance_type=INSTANCE_TYPE), role_arn=ROLE, ) source_model = mb.build(model_name=src_model_name)

rec_job = mb.generate_deployment_recommendations( tokenizer="google/gemma-4-e2b-it", concurrency=1, request_count=10, prompt_input_tokens_mean=32, output_tokens_mean=32, streaming=True, performance_target=PerformanceTarget.TTFT_MS instance_types=[INSTANCE_TYPE], advanced_optimization=False, framework=InferenceFramework.LMI, role_arn=ROLE, wait=True, )

#Comparative table across all returned recommendations print(mb.recommendations)

.best is the top-ranked row

top = mb.recommendations.best print(f"Throughput avg: {top.expected_performance.request_throughput.avg}") print(f"TTFT p99: {top.expected_performance.time_to_first_token.p99}")

auto_approve=True bypasses the ModelPackage approval-status check

rec_endpoint = mb.deploy( endpoint_name=rec_ep_name, role=ROLE, wait=True, auto_approve=True, ) print(f"Deployed: {rec_endpoint.endpoint_name} ({rec_endpoint.endpoint_status})")

Recommendation results can also be represented as a Python data frame.

import pandas as pd

pd.set_option("display.width", 200) pd.set_option("display.max_columns", None) pd.set_option("display.max_colwidth", 60)

rows = [] for i, rec in enumerate(mb.recommendations): raw = rec._raw spec = raw.model_details.inference_specification_name for m in raw.expected_performance: rows.append({ "rank": i, "spec_name": spec, "instance": raw.deployment_configuration.instance_type, "metric": m.metric, # ← attribute, not subscript "stat": m.stat, "value": float(m.value), "unit": m.unit, }) df_long = pd.DataFrame(rows) print(df_long)

rank spec_name instance metric stat value unit 0 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge RequestThroughput avg 112.7664 Requests/Second 1 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge OutputTokenThroughput avg 3608.5300 Tokens/Second 2 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge RequestLatency p50 462.1300 Milliseconds 3 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge RequestLatency p90 999.5400 Milliseconds 4 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge RequestLatency p99 1069.8400 Milliseconds 5 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge TimeToFirstToken p50 438.5300 Milliseconds 6 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge TimeToFirstToken p90 983.3300 Milliseconds 7 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge InterTokenLatency p50 0.7900 Milliseconds 8 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge InterTokenLatency p90 2.8700 Milliseconds 9 0 low-ttft-on-g6-2xlarge-lmi-26-0-0 ml.g6.2xlarge ClientSideConcurrency 64.0000 Count 10 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge RequestThroughput avg 96.8522 Requests/Second 11 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge OutputTokenThroughput avg 3099.2700 Tokens/Second 12 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge RequestLatency p50 541.1600 Milliseconds 13 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge RequestLatency p90 1122.2000 Milliseconds 14 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge RequestLatency p99 1162.5300 Milliseconds 15 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge TimeToFirstToken p50 502.9000 Milliseconds 16 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge TimeToFirstToken p90 1088.4800 Milliseconds 17 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge InterTokenLatency p50 1.0000 Milliseconds 18 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge InterTokenLatency p90 3.6100 Milliseconds 19 1 low-ttft-on-g6-2xlarge-lmi-27-0-0 ml.g6.2xlarge ClientSideConcurrency 64.0000 Count

How to interpret recommendation results

The recommendations table shows two candidate configurations (rank 0 and rank 1), both on ml.g6.2xlarge but with different LMI container versions. Here’s how to read the key metrics and choose between them:

Key metrics to compare:

RequestThroughput (avg): Requests the endpoint can serve per second. Higher is better.

OutputTokenThroughput (avg): Total tokens generated per second across all concurrent requests. Higher is better.

RequestLatency (p50/p90/p99): End-to-end time from request to full response. Lower is better.

TimeToFirstToken (p50/p90): How quickly the user sees the first streamed token. Lower is better.

InterTokenLatency (p50/p90): Delay between successive tokens during streaming. Lower is better.

Choosing between the two configurations in this example:

Rank 0 (lmi-26-0-0) delivers 112.8 req/s throughput and 3,609 tokens/s, with p90 TTFT of 983 ms and p90 latency of 1,000 ms. Rank 1 (lmi-27-0-0) delivers 96.9 req/s throughput and 3,099 tokens/s, with p90 TTFT of 1,088 ms and p90 latency of 1,122 ms. Rank 0 wins on every dimension: approximately 16% higher throughput and approximately 10 percent lower latency. The service ranks it first because the job was configured with performance_target=PerformanceTarget.TTFT_MS, meaning the optimizer prioritized configurations that minimize time-to-first-token.

General decision framework

Latency-sensitive applications (chatbots, interactive UIs): Prioritize low TTFT (p90/p99) so users perceive fast responses.

Throughput-sensitive workloads (batch summarization, offline processing): Prioritize high RequestThroughput and OutputTokenThroughput to maximize tokens per dollar.

If two configurations are close on your primary metric, use the secondary metrics as tiebreakers, then factor in cost (a smaller instance at similar performance saves money).

In this example, the top-ranked configuration (lmi-26-0-0) is the clear choice because it dominates across all metrics at the same concurrency level (64).

Deploy from previously run recommendation job

In production workflows, you often generate recommendations in one session and deploy in another. For example, a data scientist might run the recommendation job during experimentation, while an MLOps pipeline deploys the result during a release cycle. Use ModelBuilder.from_recommendation_job(job_name) to hydrate a ModelBuilder from a completed job:

from sagemaker.serve import ModelBuilder

Hydrate a fresh ModelBuilder from a completed recommendation job

mb = ModelBuilder.from_recommendation_job("my-rec-job-name") print(f"Loaded {len(mb.recommendations)} recommendations") print(mb.recommendations)

Deploy the top-ranked recommendation

endpoint = mb.deploy( role=ROLE, wait=True, auto_approve=True, )

Deploy a JumpStart model and benchmark it

After you have deployed your recommended configuration, the next step is to validate its performance under controlled conditions. Benchmarking confirms that the endpoint meets your latency and throughput requirements before serving production traffic. The SDK makes this straightforward: deploy a JumpStart model and run a synthetic load test in only a few lines of code.

import time, uuid from sagemaker.core.jumpstart.configs import JumpStartConfig from sagemaker.serve import ModelBuilder, start_benchmark from sagemaker.train.configs import Compute from sagemaker.serve import InferenceFramework, PerformanceTarget

uid = f"{int(time.time())}-{uuid.uuid4().hex[:8]}" ep_name = f"demo-bench-ep-{uid}" model_name = f"demo-bench-model-{uid}"

Build and deploy a JumpStart endpoint

mb = ModelBuilder.from_jumpstart_config( jumpstart_config=JumpStartConfig(model_id=MODEL_ID), compute=Compute(instance_type=INSTANCE_TYPE), role_arn=ROLE, ) core_model = mb.build(model_name=model_name) core_endpoint = mb.deploy(endpoint_name=ep_name)

Benchmark with a synthetic workload

job = start_benchmark( endpoint=core_endpoint, tokenizer="google/gemma-4-e2b-it", concurrency=1, request_count=10, prompt_input_tokens_mean=32, output_tokens_mean=32, streaming=True, role=ROLE, wait=True, ) result = job.show_result()

Reading benchmark results

After the benchmark completes, you need to understand whether your endpoint meets your service-level objectives. The benchmark returns a typed result object with a metrics accessor that gives you programmatic access to throughput, latency percentiles, and token-level timing. IDE autocomplete works on all fields.

Well-known shortcuts — fully typed, IDE autocomplete works

print(f"Throughput avg: {result.metrics.request_throughput.avg} req/sec") print(f"TTFT p99: {result.metrics.time_to_first_token.p99} ms") print(f"E2E latency p90: {result.metrics.request_latency.p90} ms")

Any metric AIPerf produced, by raw key

ott = result.metrics.get("output_token_throughput") if ott: print(f"Output token throughput p90: {ott.p90} {ott.unit}")

Benchmark results can similarly be represented as a Python data frame.

import pandas as pd

pd.set_option("display.width", 200) pd.set_option("display.max_columns", None) pd.set_option("display.max_colwidth", 60)

result = job.show_result()

rows = [] for name, m in result.metrics.all_metrics.items(): rows.append({ "metric": name, "unit": m.unit, "avg": m.avg, "p50": m.p50, "p90": m.p90, "p99": m.p99, }) df = pd.DataFrame(rows).set_index("metric") print(df)

unit avg p50 p90 p99 metric request_throughput requests/sec 3.841439 NaN NaN NaN request_latency ms 256.288407 206.586304 257.917252 656.735558 request_count requests 10.000000 NaN NaN NaN time_to_first_token ms 91.770545 23.977319 93.899204 636.935233 time_to_second_token ms 3.035445 3.400013 3.611624 3.670301 inter_token_latency ms 4.987234 5.518454 5.689482 5.695788 output_token_throughput tokens/sec 130.608919 NaN NaN NaN output_token_throughput_per_user tokens/sec/user 1036.753582 181.210181 1041.238702 7969.310858 output_sequence_length tokens 34.000000 34.000000 35.000000 35.000000 input_sequence_length tokens 32.000000 32.000000 32.000000 32.000000 output_token_count tokens 34.000000 34.000000 35.000000 35.000000 inter_chunk_latency ms 5.307028 5.921907 6.133938 6.599335 total_output_tokens tokens 340.000000 NaN NaN NaN benchmark_duration sec 2.603191 NaN NaN NaN total_isl tokens 320.000000 NaN NaN NaN total_osl tokens 340.000000 NaN NaN NaN http_req_sending ms 0.504841 0.230543 0.527351 2.798082 http_req_waiting ms 91.130199 23.709262 93.193301 633.795432 http_req_data_received KB 7.954883 7.956055 7.970801 7.971592 http_req_connecting ms 0.257011 0.000000 0.257011 2.338797 http_req_connection_reused ratio 0.900000 1.000000 1.000000 1.000000 http_req_blocked ms 0.000000 0.000000 0.000000 0.000000 http_req_dns_lookup ms 0.042871 0.000000 0.042871 0.390126 http_req_chunks_received count 32.600000 33.000000 34.000000 34.000000 http_req_chunks_sent count 1.000000 1.000000 1.000000 1.000000 time_to_first_output_token ms 91.770545 23.977319 93.899204 636.935233 http_req_duration ms 258.657125 207.090978 260.330959 674.649103 http_req_data_sent KB 0.270020 0.269043 0.279687 0.282852 http_req_receiving ms 167.022085 182.795457 183.395102 183.933767 e2e_output_token_throughput tokens/sec/user 152.770842 164.113321 168.852245 169.710317 osl_mismatch_diff_pct % 6.250000 6.250000 9.375000 9.375000 prefill_throughput_per_user tokens/sec/user 1183.218498 1334.595197 1370.633072 1376.052063 http_req_connection_overhead ms 0.299882 0.000000 0.299882 2.728923 http_req_total ms 258.957006 207.090978 260.630841 677.378026 osl_mismatch_count requests 8.000000 NaN NaN NaN total_token_throughput tokens/sec 253.534960 NaN NaN NaN

Benchmark metrics reference

This table breaks down the metrics reported by the benchmarking service.

Metric Description Available statistics

request_throughput Completed requests per second avg, p50, p90, p99

time_to_first_token Time from request to first streamed token avg, p50, p90, p99

request_latency End-to-end request latency avg, p50, p90, p99

output_token_throughput Output tokens per second (system-wide aggregate) avg only

Compare LMI and vLLM, then deploy the preferred option

Not sure which inference framework suits your model? Run two recommendation jobs in parallel, one for LMI and one for vLLM, compare the top results, and deploy from whichever wins.

import time, uuid from concurrent.futures import ThreadPoolExecutor, as_completed from sagemaker.core.jumpstart.configs import JumpStartConfig from sagemaker.serve import ModelBuilder from sagemaker.train.configs import Compute

def build_for(framework): bid = f"{int(time.time())}-{uuid.uuid4().hex[:6]}" mb = ModelBuilder.from_jumpstart_config( jumpstart_config=JumpStartConfig(model_id=MODEL_ID), compute=Compute(instance_type=INSTANCE_TYPE), role_arn=ROLE, ) mb.build(model_name=f"demo-fw-{framework.lower()}-{bid}") return mb

mb_lmi = build_for("LMI") mb_vllm = build_for("VLLM")

def run_rec(mb, framework): return mb.generate_deployment_recommendations( tokenizer="google/gemma-4-e2b-it", concurrency=1, request_count=10, prompt_input_tokens_mean=32, output_tokens_mean=32, streaming=True, performance_target=PerformanceTarget.TTFT_MS, instance_types=[INSTANCE_TYPE], advanced_optimization=False, framework=framework, role_arn=ROLE, wait=True, )

Run both recommendation jobs in parallel

with ThreadPoolExecutor(max_workers=2) as ex: futures = { ex.submit(run_rec, mb_lmi, "LMI"): InferenceFramework.LMI, ex.submit(run_rec, mb_vllm, "VLLM"): InferenceFramework.VLLM, } for fut in as_completed(futures): print(f"[{futures[fut]}] rec job complete.")

print("\n=== LMI ==="); print(mb_lmi.recommendations) print("\n=== vLLM ==="); print(mb_vllm.recommendations)

Pick the framework whose top recommendation has the higher throughput

winner_fw, winner_mb = max( [("LMI", mb_lmi), ("VLLM", mb_vllm)], key=lambda x: x[1].recommendations.best.expected_performance.request_throughput.avg or 0, ) print(f"Winner: {winner_fw} recipe={winner_mb.recommendations.best.recommendation_spec_name}")

fw_endpoint = winner_mb.deploy( endpoint_name=f"demo-fw-winner-ep-{uuid.uuid4().hex[:8]}", role=ROLE, wait=True, auto_approve=True, ) print(f"Deployed: {fw_endpoint.endpoint_name} ({fw_endpoint.endpoint_status})")

Sample notebook

End-to-end runnable notebooks are available on GitHub:

Notebook What it covers

pysdk-ai-inference-recommender-demo.ipynb All four scenarios end-to-end: benchmark an existing endpoint (A), get recommendations + deploy (B), replay a prior job (C), compare LMI and vLLM (D).

Clean up

To avoid ongoing charges, delete the endpoints created during this walkthrough. Run the following commands in your notebook:

import boto3 sm = boto3.client("sagemaker")

Delete benchmark endpoint

sm.delete_endpoint(EndpointName=ep_name) sm.delete_endpoint_config(EndpointName=ep_name)

Delete recommendation endpoint

sm.delete_endpoint(EndpointName=rec_ep_name) sm.delete_endpoint_config(EndpointName=rec_ep_name)

For pricing details on Amazon SageMaker real-time inference instances, see Amazon SageMaker AI Pricing.

Conclusion

In this post, you walked through the complete inference optimization journey. You generated deployment recommendations that explore instance and framework configurations, interpreted the ranked results, and deployed the optimal configuration. You also validated performance with benchmarks under realistic load and compared serving frameworks head-to-head. With the Amazon SageMaker Python SDK integration for generative AI inference recommendations, this entire workflow lives in a single notebook, which removes the context-switching between console UIs and API calls.

To get started, upgrade to Amazon SageMaker Python SDK v3 and explore the sample notebook included with this post.

Additional resources

Generative AI inference recommendations in Amazon SageMaker AI.

Generative AI inference benchmarks in Amazon SageMaker AI.

SageMaker Python SDK documentation.

Getting started with SageMaker JumpStart.

Sample notebooks on GitHub.

Have feedback or questions? Let us know in the Amazon SageMaker discussion forums.

About the authors