Skip to content
AI News HubLIVE
Original source2 min read

Theoretical Limits of Language Model Alignment

Summary

This paper characterizes the information-theoretic limits of KL-regularized alignment, deriving a closed-form expression for optimal reward improvement governed by Jeffreys divergence rather than √KL, and shows best-of-N approaches the limit while PPO and GRPO are suboptimal.

Theoretical Limits of Language Model Alignment
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

content type paperpublished November 2026

Theoretical Limits of Language Model Alignment

AuthorsLucas Monteiro Paes*, Natalie Mackraz*, Barry-John Theobald, Federico Danieli

View publication

Language model (LM) alignment improves model outputs to reflect human preferences while preserving the capabilities of the base model. The most common alignment approaches are (i) reinforcement learning, which maximizes the expected reward under a KL-divergence constraint, and (ii) best-of-NNN alignment, which selects the highest-reward output among NNN independent samples. Despite their widespread use, the fundamental limits of reward improvement under a KL budget remain poorly understood. We characterize the information-theoretic limits of KL-regularized alignment by deriving the maximum achievable expected reward gain for a fixed KL-divergence budget. Our first result provides a closed-form expression for the optimal reward improvement, governed by a Jeffreys divergence term rather than the √KL√KL√KL used in prior analyses. We further reformulate this expression as a covariance under the base model, yielding a practical estimator that predicts achievable alignment gains from base model samples alone. We extend our analysis to the proxy reward setting, showing that the gap between ideal and proxy alignment (reward hacking) grows with the magnitude of reward error and when the KL penalty factor decreases. We then prove that reward ensembling mitigates reward hacking, providing a theoretical justification for this technique used in practice. Empirically, we compute the KL–reward Pareto frontier for two tasks for LMs, safety and summarization, and show that best-of-NNN closely approaches the theoretical limit, while PPO and GRPO remain substantially suboptimal. Our theoretical results shed light on several empirically observed phenomena in the alignment literature and suggest that algorithmic improvements are needed to achieve optimal alignment without high inference costs.

  • Equal contribution

SALSA: Soup-Based Alignment Learning for Stronger Adaptation in RLHF

December 5, 2024research area Speech and Natural Language ProcessingWorkshop at NeurIPS

This paper was accepted at the Fine-Tuning in Modern Machine Learning (FITML) Workshop at NeurIPS 2024.

In Large Language Model (LLM) development, Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning models with human values and preferences. RLHF traditionally relies on the Kullback-Leibler (KL) divergence between the current policy and a frozen initial policy as a reference, which is added as a penalty in policy…

Read more

On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization

October 9, 2024research area Methods and Algorithms, research area Speech and Natural Language Processingconference EMNLP

Reinforcement Learning from Human Feedback (RLHF) is an effective approach for aligning language models to human preferences. Central to RLHF is learning a reward function for scoring human preferences. Two main approaches for learning a reward model are 1) training an explicit reward model as in RLHF, and 2) using an implicit reward learned from preference data through methods such as Direct Preference Optimization (DPO). Prior work has shown…

Read more

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • The optimal reward improvement is governed by a Jeffreys divergence term, not the √KL used in prior analyses.
  • The gap between ideal and proxy alignment (reward hacking) grows with reward error and decreases with the KL penalty factor.
  • Reward ensembling mitigates reward hacking, providing theoretical justification for this practice.
  • Empirically, best-of-N closely approaches the theoretical limit, while PPO and GRPO remain substantially suboptimal.

Highlights and analysis are generated automatically and may contain errors. Check the original source.