Modern machine learning is trained by stochastic gradient descent (SGD), whose performance critically depends on how the learning rate (LR) is adjusted and decreased over time. Yet existing LR regimes may be intricate, or need to tune one or more additional hyper-parameters manually whose bottlenecks include huge computational expenditure, time and power in practice. This work, in a natural and direct manner, clarifies how LR should be updated automatically only according to the intrinsic variation of stochastic gradients. An explainable LR regime by leveraging stochastic second-order algorithms is developed, behaving a similar pattern to heuristic algorithms but implemented simply without any parameter tuning requirement, where it is of an automatic procedure that LR should increase (decrease) as the norm of stochastic gradients decreases (increases). The resulting LR regime shows its efficiency, robustness, and scalability in different classical stochastic algorithms, containing SGD, SGDM, and SIGNSGD, on machine learning tasks.

Curriculum analytics (CA) studies curriculum structure and student data to ensure the quality of educational programs. An essential aspect is studying course properties, which involves assigning each course a representative difficulty value. This is critical for several aspects of CA, such as quality control (e.g., monitoring variations over time), course comparisons (e.g., articulation), and course recommendation (e.g., advising). Measuring course difficulty requires careful consideration of multiple factors: First, when difficulty measures are sensitive to the performance level of enrolled students, it can bias interpretations by overlooking student diversity. By assessing difficulty independently of enrolled students' performances, we can reduce the risk of bias and enable fair, representative assessments of difficulty. Second, from a measurement theoretic perspective, the measurement must be reliable and valid to provide a robust basis for subsequent analyses. Third, difficulty measures should account for covariates, such as the characteristics of individual students within a diverse populations (e.g., transfer status). In recent years, various notions of difficulty have been proposed. This paper provides the first comprehensive review and comparison of existing approaches for assessing course difficulty based on grade point averages and latent trait modeling. It further offers a hands-on tutorial on model selection, assumption checking, and practical CA applications. These applications include monitoring course difficulty over time and detecting courses with disparate outcomes between distinct groups of students (e.g., dropouts vs. graduates), ultimately aiming to promote high-quality, fair, and equitable learning experiences. To support further research and application, we provide an open-source software package and artificial datasets, facilitating reproducibility and adoption.

Large language models (LLMs) exhibit impressive in-context learning (ICL) abilities, enabling them to solve wide range of tasks via textual prompts alone. As these capabilities advance, the range of applicable domains continues to expand significantly. However, identifying the precise mechanisms or internal structures within LLMs that allow successful ICL across diverse, distinct classes of tasks remains elusive. Physics-based tasks offer a promising testbed for probing this challenge. Unlike synthetic sequences such as basic arithmetic or symbolic equations, physical systems provide experimentally controllable, real-world data based on structured dynamics grounded in fundamental principles. This makes them particularly suitable for studying the emergent reasoning behaviors of LLMs in a realistic yet tractable setting. Here, we mechanistically investigate the ICL ability of LLMs, especially focusing on their ability to reason about physics. Using a dynamics forecasting task in physical systems as a proxy, we evaluate whether LLMs can learn physics in context. We first show that the performance of dynamics forecasting in context improves with longer input contexts. To uncover how such capability emerges in LLMs, we analyze the model's residual stream activations using sparse autoencoders (SAEs). Our experiments reveal that the features captured by SAEs correlate with key physical variables, such as energy. These findings demonstrate that meaningful physical concepts are encoded within LLMs during in-context learning. In sum, our work provides a novel case study that broadens our understanding of how LLMs learn in context.

Neural Information Processing Systems http://nips.cc/

Decision trees offer this inter-pretability, especially when they are small.

The core of our algorithm is the computation of the stationary distribution of a Markov chain defined on an item-item graph.

This yields continuous-time counterparts of Fast Weight Programmers and linear Transformers.

However, in these domains, the number of available problems typically drops rapidly as a function of problem length (e.g. Figure 2).

The softened softmax probability calculated without true-class logit on the server/client modelClass Distribution on Datasets: p " r p For the federated learning situation, we calculate this measure on the server model. Here we provide details of our experimental setups. Multi-GPU training is not conducted in the paper experiments. The details about each datasets and setups are described in Table 4. CIFAR-100, we add Cutout [12] augmentation. Details datasets setups used in the experiment. We use a momentum SGD optimizer with an initial learning rate of 0.01, and the momentum is set as The learning rate is decayed with a factor of 0.99 at each round, and In the motivational experiment in Section 3, we fix the learning rate as 0.01. Since we assume a synchronized federated learning scenario, parallel distributed learning is simulated by sequentially training the sampled clients and then aggregating them as a global model. For the implemented algorithms, we search hyperparameters and choose the best among the candidates. The hyperparameters for each algorithm is in Table 5. Sharding strategy, and the size of local datasets is identical. The conceptual illustration of federated distillation methods is in Figure 9. On the other hand, our proposed FedNTD does not have such constraints (Figure 9c). Additional resource requirements compared to FedAvg.Method No Additional Requirements on: Statefulness? We extend the motivational experiment in Section 3.1 to the main experimental setups. The value in the parenthesis is the forgetting F . The value in the parenthesis is the forgetting F . The value in the parenthesis is the forgetting F . The value in the parenthesis is the forgetting F . We report an additional experiment on popular architecture, ResNet-10. ResNet-10 is about 10x larger than the 2-conv + 2-fc model for the main experiments. The result is plotted in Figure 11. Here we investigate the personalized performance of our FedNTD. The results are in Table 13 and Figure 13 shows their corresponding learning curves. FedNTD consistently improves the performance even in such cases.Figure 13: Learning curves that corresponds to Table 13. The introduced loss term of FedAlign aims to seek out-of-distribution generality w.r.t. Figure 14: Loss space of learned model (Client 16 / LDA α " 0. 5).

In federated learning, a strong global model is collaboratively learned by aggregating clients' locally trained models.