Why Transformers Need Adam: A Hessian Perspective

Feb-18-2026, 15:12:37 GMT–Neural Information Processing Systems

CNNs, MLPs, and quadratic problems, and find that SGD can perform on par with Adam on problems without block heterogeneity, but performs worse than Adam when the heterogeneity exists.

large language model, machine learning, natural language, (20 more...)

Neural Information Processing Systems

Feb-18-2026, 15:12:37 GMT

Conferences PDF

Country:
- Asia > China
  - Guangdong Province > Shenzhen (0.04)
  - Hong Kong (0.04)
- Europe > Germany
  - Bavaria > Upper Bavaria > Munich (0.04)

Genre:
- Research Report
  - Experimental Study (0.93)
  - New Finding (0.92)

Technology:
- Information Technology > Artificial Intelligence
  - Machine Learning > Neural Networks
    - Deep Learning (1.00)
  - Natural Language > Large Language Model (0.94)
  - Representation & Reasoning > Optimization (0.92)
  - Vision (1.00)