Optimization
Stochastic Spectral and Conjugate Descent Methods
Dmitry Kovalev, Peter Richtarik, Eduard Gorbunov, Elnur Gasanov
Neural Information Processing Systems http://nips.cc/
H3T: Efficient Integration of Memory Optimization and Parallelism for High-Throughput Transformer Training Y uzhong Wang
Transformer-based models, their huge parameter size poses a serious challenge to their training, both from the storage and computation perspectives.
DAGs with NO TEARS: Continuous Optimization for Structure Learning
Xun Zheng, Bryon Aragam, Pradeep K. Ravikumar, Eric P. Xing
Neural Information Processing Systems http://nips.cc/
Learning Signed Determinantal Point Processes through the Principal Minor Assignment Problem
Neural Information Processing Systems http://nips.cc/
Statistical Model Aggregation via Parameter Matching
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang
We consider the problem of aggregating models learned from sequestered, possibly heterogeneous datasets.
Learning to Optimize in Swarms
Yue Cao, Tianlong Chen, Zhangyang Wang, Yang Shen
Neural Information Processing Systems http://nips.cc/
Acceleration through Optimistic No-Regret Dynamics
Jun-Kun Wang, Jacob D. Abernethy
Zero-sum games can be solved using online learning dynamics, where a classical technique involves simulating two no-regret algorithms that play against each other and, afterT rounds, the average iterate is guaranteed to solve the original optimization problem with error decaying asO(logT/T). In this paper we show that the technique can be enhanced to a rate ofO(1/T2) by extending recent work [22, 25] that leverages optimistic learning to speed upequilibrium computation.
The Randomized Midpoint Method for Log-Concave Sampling
Neural Information Processing Systems http://nips.cc/
GraphStructuredPredictionEnergyNetworks
Specifically,GSPENs combine thecapabilities ofclassicalstructured prediction models andSPENs andhavetheability toexplicitly model localstructure whenknown or assumed, while providing the ability to learn an unknown or more global structure implicitly.