Fine-tune NVIDIA Nemotron 3 models with Amazon SageMaker AI serverless model customization

Model customization transforms general-purpose AI models into specialized enterprise assets. By fine-tuning foundation models (FMs) on domain-specific data, businesses teach AI their unique workflows, terminology, and deep domain specialization, along with strict adherence to brand voice and fewer hallucinations. For enterprises, this is more than an optimization. It’s the creation of proprietary intellectual property. A fine-tuned model encodes an organization’s unique intelligence and best practices into its architecture. This builds a competitive advantage that is difficult to replicate with off-the-shelf public frontier models. At the same time, fine-tuning smaller, open-weight models on targeted tasks often matches or exceeds the performance of much larger proprietary models. This approach delivers significant cost savings while keeping sensitive data within secure, private infrastructure.

Amazon SageMaker AI offers a wide selection of open source models and fine-tuning techniques to help organizations tailor foundation models to their unique needs. Now, SageMaker AI introduces serverless model customization for NVIDIA Nemotron 3 models, starting with Nemotron 3 Nano (30B total parameters, 3B active) and Nemotron 3 Super (120B total parameters, 12B active). With supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and reinforcement learning with AI feedback (RLAIF), you can adapt these high-performance open-weight models to your specific domains and workflows without provisioning or managing any infrastructure. For a complete list of open models available for serverless model customization, see Customize open weight models in the Amazon SageMaker AI documentation.

In this post, we explore what makes the Nemotron 3 architecture unique, walk through the fine-tuning techniques available, and show you step-by-step how to get started with serverless customization using SageMaker Studio.

Overview of NVIDIA Nemotron 3 models on Amazon SageMaker AI

NVIDIA Nemotron 3 is a family of open-weight large language models (LLMs) built on a hybrid Mamba-Transformer Mixture-of-Experts (MoE) architecture with native support for up to 1M-token context lengths. The architecture interleaves three complementary layer types: Mamba-2 layers for efficient linear-time sequence processing, Transformer attention layers for precise associative recall, and Latent Mixture-of-Experts (LatentMoE) layers that compress tokens before routing to specialized experts. This design activates only a fraction of total parameters per forward pass (for example, 12B of 120B in the Super variant), delivering high throughput and strong accuracy at significantly lower compute cost. The models use multi-environment reinforcement learning through NeMo Gym, which aligns them to real-world, multi-step agentic tasks across domains such as coding, reasoning, and long-context analysis.

Nemotron 3 Nano 30B

Nemotron 3 Nano is a small language model optimized for high compute efficiency while maintaining strong accuracy on specialized tasks. Nemotron 3 Nano performs strongly on coding and reasoning tasks among open language models in its size class. Trained using multi-environment reinforcement learning through NeMo Gym, the model achieves 4x higher throughput than its predecessor Nemotron 2 Nano. Its efficient 3B active parameter footprint makes it ideal for high-volume, multi-agent workloads where cost and latency matter. For a deeper look at the architecture and training techniques, see the NVIDIA developer blog.

Nemotron 3 Super 120B

Nemotron 3 Super is a larger model designed for high-efficiency multi-agent AI and complex reasoning tasks that require more capacity than Nano while maintaining cost efficiency. Nemotron 3 Super delivers high compute efficiency, throughput, and accuracy for complex multi-agent applications such as software development and cybersecurity triaging. The model performs well at reasoning, coding, and long-context analysis, while remaining efficient enough to run continuously at scale. This makes it a good fit for IT ticket automation, enterprise workflow orchestration, and autonomous agent systems that require sustained multi-step reasoning. For more details, see the NVIDIA developer blog on Nemotron 3 Super.

SageMaker AI serverless model customization

Amazon SageMaker AI serverless model customization removes the undifferentiated heavy lifting of fine-tuning. You don’t need to provision GPU clusters, configure distributed training frameworks, or manage checkpointing and fault tolerance. SageMaker AI handles infrastructure provisioning and training orchestration, so you can focus on your data, business use case, and evaluation, and pay only for what you use. You can learn more about SageMaker AI serverless model customization in the AWS documentation.

For Nemotron 3 models, SageMaker AI serverless model customization supports the Supervised Fine-Tuning (SFT), Reinforcement Learning with Verifiable Rewards (RLVR) and Reinforcement Learning from AI Feedback (RLAIF) fine-tuning techniques.

Technique Description Best For

Supervised Fine-Tuning (SFT) Provide labeled input-output pairs to teach the model new behaviors. High-quality examples of the behavior you want: domain Q&A pairs, formatted tool calls, style-aligned responses, or task-specific instruction completions

Reinforcement Fine-Tuning (RFT / RLVR) Use Reinforcement Learning with Verifiable Rewards (RLVR) to optimize model behavior against a reward signal. The model generates multiple candidate responses per prompt, a reward function scores them, and the model updates its policy to favor what works. Tasks with naturally verifiable objectives like tool calling accuracy, code correctness, or format compliance

Reinforcement Learning from AI Feedback (RLAIF) Use a separate AI model to guide the model optimization. An AI model evaluates model outputs and provides feedback signals, which supports iterative policy improvement without human-labeled reward data. Aligning model tone, helpfulness, and safety; improving response quality when human evaluation is expensive or subjective; refining open-ended generation tasks

Let’s walk through how to get started with serverless model customization for Nemotron 3 models. While the base Nemotron 3 models deliver strong general-purpose performance, enterprise use cases need domain-specific behavior that base models alone cannot achieve. With model customization, you can adapt these models for industry-specific terminology and decision patterns, train reliable tool calling with your organization’s APIs, align outputs with your brand voice, refine multi-step agentic reasoning for your architectures, and optimize cost by specializing the smaller Nano model to match larger model performance on targeted tasks.

Getting started with SageMaker AI serverless model customization

You can get started with serverless model customization through the Amazon SageMaker Studio console or programmatically using the SageMaker Python SDK. On the console, navigate to the Models page, select your Nemotron 3 model, and follow the guided workflow to configure your training data and launch a customization job. Alternatively, if you’re already working within SageMaker AI, you can use the agentic functionality with agent skills to accelerate your model customization workflow. The following sections walk you through the prerequisites, data preparation, and step-by-step instructions using the SageMaker Studio console. For a detailed programmatic example with the SageMaker Python SDK for customizing an open-source model, see the AWS samples GitHub repository.

Prerequisites

Before you begin, verify that you have:

An AWS account with AWS Identity and Access Management (IAM) permissions for Amazon SageMaker AI.

A SageMaker AI domain with Studio access.

Your training data in the required structure and format.

Prepare your training data for SageMaker AI serverless model customization

High-quality training data is the foundation of any successful fine-tuning job. For serverless model customization on SageMaker AI, your data must be formatted as JSONL (JSON Lines), where each line represents a single training example. The specific schema depends on the technique you choose: SFT requires conversation-format examples with labeled input-output pairs, while RFT (RLVR) requires prompts paired with ground truth values for your reward function. Properly structured data ensures the model learns the behaviors you intend without introducing noise or formatting errors. For a hands-on walkthrough of preparing your training data, see the Data Preparation module in the SageMaker AI serverless model customization workshop. Alternatively, if you are working with SageMaker AI, you can use the built-in coding agent with agent skills to automatically prepare and validate your data formatting, reducing manual effort and helping you get to training faster.

Model customization in SageMaker AI Studio

Follow these steps to customize a Nemotron 3 model using the SageMaker AI Studio console.

Open Amazon SageMaker AI Studio and in the left navigation pane, choose Models.

Navigate to the model you want to customize in the UI. Search for “NVIDIA” to find the Nemotron 3 family of models, and select the NVIDIA model that you want (NVIDIA-Nemotron-3-Nano-30B-* or NVIDIA-Nemotron-3-Super-120B-*) for the next step.

Select your model customization technique from the supported Supervised Fine-Tuning (SFT), Reinforcement Learning with Verifiable Rewards (RLVR) and Reinforcement Learning from AI Feedback (RLAIF) fine-tuning techniques. When choosing a reward function type for RLVR, consider your task requirements. The built-in reward function (Exact Match, Code Execution, Math Answers) works well for tasks with single, objectively correct answers, requiring no additional code. Choose a custom reward function when your task needs richer scoring logic, such as partial credit, format checks, reasoning quality evaluation, or domain-specific rules. With custom reward functions, you can score on multiple signals, shape rewards to avoid all-zero gradients on early rollouts, emit observability metrics, and encode the Python verification logic your task requires. For detailed guidance on authoring and registering a custom reward function, see the RLVR workshop documentation.

Configure your training data by selecting an existing dataset (if available) or creating a new dataset (see the preceding section for information about preparing your dataset).

Set the customization hyperparameters or use recommended defaults.

Choose Submit to launch the model customization job.

SageMaker AI automatically provisions the required compute, executes the training job, and captures continuous logs. The training metrics are automatically logged to the SageMaker MLflow App by default for training tracking.

Monitor training progress

You can monitor the status on the model home page, which displays training performance, as shown in the following screenshot. Several high-level metrics are worth monitoring. Train Reward (for RLVR) should increase steadily. Training Loss and Validation Loss should decrease and track generalization, respectively. Policy Entropy (for RLVR) decreases as the model gains confidence. Gradient Norm should stabilize to indicate convergence.

The detailed training and validation metrics are also logged to the associated SageMaker AI MLflow App, as shown in the following screenshot. This captures a comprehensive set of metrics and parameters that track training progress, and model behavior. In the MLflow tracking UI, these metrics are organized by the component they measure (actor, critic, rollout, performance), so you can diagnose training health at a glance.

Evaluate your fine-tuned model

After training completes, you can evaluate the fine-tuned model using the built-in evaluation features of SageMaker AI serverless model customization. It provides three methods to assess the quality of your customized model, as shown in the following screenshot. LLM-as-a-Judge uses an Amazon Bedrock frontier model to grade responses against quality metrics without requiring ground-truth labels. Custom Scorer applies your own reward functions or built-in scorers to produce standard natural language processing (NLP) metrics such as F1, ROUGE, and BLEU. Benchmarks scores your model on standardized academic benchmarks (MMLU, BBH, GPQA, MATH, IFEval) for broad capability assessment across reasoning, knowledge, and instruction-following.

You can also turn on Compare with base model in evaluation to directly measure how your post-trained model performs relative to the base model. Together with the previous training metrics, MLflow tracks the training dynamics (rewards, KL divergence, loss). The evaluation measures output quality from an end-user perspective, giving you a complete picture of the model fine-tuning effectiveness.

Deploy the fine-tuned model

Deploy your customized model directly from the model details page on the console. You can also deploy to SageMaker Inference endpoints, or you can download model weights from an Amazon Simple Storage Service (Amazon S3) bucket for self-managed deployment. The deployment options auto-populate defaults, giving you full flexibility over compute and scaling based on your traffic and throughput requirements. The following screenshot shows the deployment of the fine-tuned NVIDIA Nemotron Nano 30B using an ml.g6e instance powered by NVIDIA L40S Tensor Core GPUs. The deployment uses SageMaker inference components and, by default, serves the merged model weights, where the base model and LoRA adapter are combined into a single set of weights for optimized inference. Because this is a LoRA fine-tune, you can also self-host and serve the unmerged LoRA adapter separately, because you have access to both the base weights and the adapter weights in your S3 bucket. After deployment, you invoke the endpoint using the invoke method with the AWS Command Line Interface (AWS CLI) or SDK.

Clean up

To avoid incurring unnecessary charges, we recommend deleting your SageMaker AI Studio domain, SageMaker Endpoints, and any other resources that you created after you’re done using them. The specific cost of using SageMaker AI serverless model customization depends on the base model you choose and the customization stage. See the Amazon SageMaker AI pricing page for the cost breakdown and details.

Conclusion

With serverless model customization for NVIDIA Nemotron 3 models on Amazon SageMaker AI, you can now adapt these high-performance open-weight models to your specific domains and workflows. Whether you’re fine-tuning Nemotron 3 Nano for cost-efficient agentic task execution or customizing Nemotron 3 Super for complex multi-agent orchestration, SageMaker AI handles compute provisioning, training orchestration, and metric tracking so you can focus on your data, evaluation, and deployment.

Get started today with serverless Model Customization on Amazon SageMaker AI. For detailed examples of customizing open-source models, see the AWS samples GitHub repository. To learn more, see the Amazon SageMaker AI model customization documentation.

About the authors