These authors contributed equally to this work. Noisy labels pose a substantial challenge in machine learning, often resulting in overfitting and poor generalization. Sharpness-Aware Minimization (SAM), as demonstrated by Foret et al. (2021), improves generalization over traditional Stochastic Gradient Descent (SGD) in classification tasks with noisy labels by implicitly slowing noisy learning. While SAM's ability to generalize in noisy environments has been studied in several simplified settings, its full potential in more realistic training settings remains underexplored. In this work, we analyze SAM's behavior at each iteration, identifying specific components of the gradient vector that contribute significantly to its robustness against noisy labels. Based on these insights, we propose SANER (Sharpness-Aware Noise-Explicit Reweighting), an effective variant that enhances SAM's ability to manage noisy fitting rate. Our experiments on CIFAR-10, CIFAR-100, and Mini-WebVision demonstrate that SANER consistently outperforms SAM, achieving up to an 8% increase on CIFAR-100 with 50% label noise. The issue of noisy labels due to human error annotation has been commonly observed in many largescale datasets such as CIFAR-10N, CIFAR-100N (Wei et al., 2022), Clothing1M (Xiao et al., 2015), and WebVision (Li et al., 2017). Over-parameterized deep neural networks, which have enough capacity to memorize entire large datasets, can easily overfit such noisy label data, leading to poor generalization performance (Zhang et al., 2021). Moreover, the lottery ticket hypothesis (Frankle & Carbin, 2019) indicates that only a subset of the network's parameters is crucial for generalization. This highlights the importance of noise-robust learning, where the goal is to train a robust classifier despite the presence of inaccurate or noisy labels in the training dataset. Sharpness-Aware Minimization (SAM), introduced by Foret et al. (2021), is an optimizer designed to find better generalization by searching for flat minima. It has shown superior performance over SGD in various tasks, especially in classification tasks involving noisy labels Baek et al. (2024). Understanding the mechanisms behind the success of SAM is crucial for further improvements in handling label noise.

Therefore, we split the action prediction problem into two steps: (1) we predict a Most Relevant Point, or MRP, which tells us which region of the world the policy must attend to; and (2) we reactively predict actions which determine where the robot should move in relation to that MRP: for example, how to approach the handle of an oven and when to close the gripper to grasp it. These two operations are performed sequentially using a modified PointNet++ (Qi et al., 2017) model that we refer to as Attention-based PointNet (A-PointNet), shown in Figure 2. The MRP Predictor can then be agnostic to the position of the robot, instead focusing on the features of the object relevant to the overall task, while the Action Predictor can learn to focus on features relevant just to what the next action should be. For example, in Figure 7, the MRP Predictor learns to focus on the handle; the Action Predictor focuses on the angle of the oven door. Image Pre-Processing First we convert the RGB and depth images into a point cloud. We augment the point cloud of the current timestep with our context c, the point cloud from the beginning of the episode. This aids both in combating occlusion, as well as in disambiguating between similar observations that occur during different trajectories. To reduce compute, we crop the working area to 1m, and down-sample using grid pooling, with a resolution of 1cm for the current timestep and 2.5cm for the context. Specifically, we select a random point in each voxel, to reduce overfitting.