Deep Learning
Topological Generalization Bounds for Discrete-Time Stochastic Optimization Algorithms
Recent studies have revealed that properties of training trajectories can be indicative of generalization.
Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers
Interestingly, the scaling performance of structured matrices is explored, revealing steeper curves in scaling training FLOPs, along with a favorable scaling trend in the overtraining regime. Specifically, we show that wide and structured networks can utilize training FLOPs more efficiently, with fewer parameters and lower loss than dense models at their optimal trade-off.
Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning Hang Zhou 1,2, Yehui Tang
Unfortunately, collecting high-quality and diverse data is both expensive and time-consuming. To mitigate this issue, we propose a novel Star-Agents framework, which automates the enhancement of data quality across datasets through multi-agent collaboration and assessment. The framework adopts a three-pronged strategy.
Weight decay induces low-rank attention layers
The effect of regularizers such as weight decay when training deep neural networks is not well understood.
Explicit Eigenvalue Regularization Improves Sharpness-A ware Minimization
Sharpness-A ware Minimization (SAM) has attracted significant attention for its effectiveness in improving generalization across various tasks. However, its underlying principles remain poorly understood. In this work, we analyze SAM's
Revisiting Score Propagation in Graph Out-of-Distribution Detection Longfei Ma
OOD scores along the graph structure (as shown in Figure 1).
Atharva Mete
We compare to state-of-the-art imitation learning and L VM baselines and see that QueST's architecture leads to strong performance on several multitask and few-shot learning benchmarks.
