Draft:Sharpness aware minimization Source: en.wikipedia.org/wiki/Draft:Sharpness_aware_minimization

This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these messages) This article may be too technical for most readers to understand. Please help improve it to make it understandable to non-experts, without removing the technical details. (June 2025) (Learn how and when to remove this message) This article may incorporate text from a large language model. Such text potentially includes hallucinated information or fictitious references. Copyright violations or claims lacking verification must be removed. Please see the associated project page for additional guidance. (June 2025) This article is an orphan, as no other articles link to it. Please introduce links to this page from related articles; try the Find link tool for suggestions. (June 2025) (Learn how and when to remove this message)

Sharpness Aware Minimization (SAM) is an optimization algorithm used in machine learning that aims to improve model generalization. The method seeks to find model parameters that are located in regions of the loss landscape with uniformly low loss values, rather than parameters that only achieve a minimal loss value at a single point. This approach is described as finding "flat" minima instead of "sharp" ones. The rationale is that models trained this way are less sensitive to variations between training and test data, which can lead to better performance on unseen data.^[1]

The algorithm was introduced in a 2020 paper by a team of researchers including Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.^[1]

SAM modifies the standard training objective by minimizing a "sharpness-aware" loss. This is formulated as a minimax problem where the inner objective seeks to find the highest loss value in the immediate neighborhood of the current model weights, and the outer objective minimizes this value:^[1]

${\displaystyle \min _{w}\max _{\|\epsilon \|_{p}\leq \rho }L_{\text{train}}(w+\epsilon )+\lambda \|w\|_{2}^{2}}$

In this formulation:

• ${\displaystyle w}$ represents the model's parameters (weights).
• ${\displaystyle L_{\text{train}}}$ is the loss calculated on the training data.
• ${\displaystyle \epsilon }$ is a perturbation applied to the weights.
• ${\displaystyle \rho }$ is a hyperparameter that defines the radius of the neighborhood (an ${\displaystyle L_{p}}$ ball) to search for the highest loss.
• An optional L2 regularization term, scaled by ${\displaystyle \lambda }$, can be included.

A direct solution to the inner maximization problem is computationally expensive. SAM approximates it by taking a single gradient ascent step to find the perturbation ${\displaystyle \epsilon }$. This is calculated as:^[1]

${\displaystyle \epsilon (w)=\rho {\frac {\nabla L_{\text{train}}(w)}{\|\nabla L_{\text{train}}(w)\|_{2}}}}$

The optimization process for each training step involves two stages. First, an "ascent step" computes a perturbed set of weights, ${\displaystyle w_{\text{adv}}=w+\epsilon (w)}$, by moving towards the direction of the highest local loss. Second, a "descent step" updates the original weights ${\displaystyle w}$ using the gradient calculated at these perturbed weights, ${\displaystyle \nabla L_{\text{train}}(w_{\text{adv}})}$. This update is typically performed using a standard optimizer like SGD or Adam.^[1]

SAM has been applied in various machine learning contexts, primarily in computer vision. Research has shown it can improve generalization performance in models such as Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) on image datasets including ImageNet, CIFAR-10, and CIFAR-100.^[1]

The algorithm has also been found to be effective in training models with noisy labels, where it performs comparably to methods designed specifically for this problem.^[2] Some studies indicate that SAM and its variants can improve out-of-distribution (OOD) generalization, which is a model's ability to perform well on data from distributions not seen during training. Other areas where it has been applied include gradual domain adaptation and mitigating overfitting in scenarios with repeated exposure to training examples.^[1]

A primary limitation of SAM is its computational cost. By requiring two gradient computations (one for the ascent and one for the descent) per optimization step, it approximately doubles the training time compared to standard optimizers.^[1]

The theoretical convergence properties of SAM are still under investigation. Some research suggests that with a constant step size, SAM may not converge to a stationary point.^[3] The accuracy of the single gradient step approximation for finding the worst-case perturbation may also decrease during the training process.^[4]

The effectiveness of SAM can also be domain-dependent. While it has shown benefits for computer vision tasks, its impact on other areas, such as GPT-style language models where each training example is seen only once, has been reported as limited in some studies. Furthermore, while SAM seeks flat minima, some research suggests that not all flat minima necessarily lead to good generalization. The algorithm also introduces the neighborhood size ${\displaystyle \rho }$ as a new hyperparameter, which requires tuning.^[1]

Active research on SAM focuses on reducing its computational overhead and improving its performance. Several variants have been proposed to make the algorithm more efficient. These include methods that attempt to parallelize the two gradient computations, apply the perturbation to only a subset of parameters, or reduce the number of computation steps required.^[5][6][7] Other approaches use historical gradient information or apply SAM steps intermittently to lower the computational burden.^[8]

To improve performance and robustness, variants have been developed that adapt the neighborhood size based on model parameter scales (Adaptive SAM or ASAM)^[4] or incorporate information about the curvature of the loss landscape (Curvature Regularized SAM or CR-SAM). Other research explores refining the perturbation step by focusing on specific components of the gradient or combining SAM with techniques like random smoothing.^[9][10]

Theoretical work continues to analyze the algorithm's behavior, including its implicit bias towards flatter minima and the development of broader frameworks for sharpness-aware optimization that use different measures of sharpness.

This article has not been added to any content categories. Please help out by adding categories to it so that it can be listed with similar articles. (June 2025)