Stochastic Gradient Descent (SGD’s) Frequency Bias and How Adam Fixes It

Modern language models face an optimization challenge due to uneven token distributions: parameters associated with common tokens receive frequent gradient updates while rare token parameters may go hundreds of steps without a meaningful signal. Standard SGD uses the same learning rate for all parameters, causing rare token weights to remain near initialization. Adam's variance normalization automatically gives larger effective learning rates to underrepresented features, enabling them to converge quickly. A controlled NumPy experiment with a six-token vocabulary spanning four orders of magnitude in frequency shows that SGD yields weights near 0.15 for the rarest token (true weight 1.0), while Adam keeps all rare token weights close to 1.0.