Goto

Collaborating Authors

 Statistical Learning


Multi-task Online Learning for Probabilistic Load Forecasting

arXiv.org Machine Learning

Load forecasting is essential for the efficient, reliable, and cost-effective management of power systems. Load forecasting performance can be improved by learning the similarities among multiple entities (e.g., regions, buildings). Techniques based on multi-task learning obtain predictions by leveraging consumption patterns from the historical load demand of multiple entities and their relationships. However, existing techniques cannot effectively assess inherent uncertainties in load demand or account for dynamic changes in consumption patterns. This paper proposes a multi-task learning technique for online and probabilistic load forecasting. This technique provides accurate probabilistic predictions for the loads of multiple entities by leveraging their dynamic similarities. The method's performance is evaluated using datasets that register the load demand of multiple entities and contain diverse and dynamic consumption patterns. The experimental results show that the proposed method can significantly enhance the effectiveness of current multi-task learning approaches across a wide variety of load consumption scenarios.


Reviews: Leader Stochastic Gradient Descent for Distributed Training of Deep Learning Models

Neural Information Processing Systems

The paper proposes and theoretically analyzes a distributed SGD algorithm where the workers are pulled towards the best performing worker rather than the average worker. All three reviewers consider the theoretical contribution (analysis of convergence and cost of communication) to be interesting and rigorous. At the same time, one reviewer feels the theoretical analysis applies to a simplified case and may not shed light on the experiments that are done in more complex settings. The reviewer's were not satisfied by the rebuttal, but maintained that the paper is publishable. Overall, there is a consensus that is is a fine paper and I recommend acceptance.


Review for NeurIPS paper: Robustness Analysis of Non-Convex Stochastic Gradient Descent using Biased Expectations

Neural Information Processing Systems

Weaknesses: While the "biased expectation" appears to be a powerful tool, the overall results are restricted to the gradients of the algorithm at _some_ time t in the last T iterates. While this is a common outcome of the standard analysis of SGD, it would be nice if (with some additional assumptions on f) the results could be transposed to f(x_t) or x_t within some basin of attraction. The special case of s 0 needs much more detailed treatment. While the authors point out in the supplement that \phi is continuous at s 0, much of the document switches between looking at s- 0 or s 0 without explanation. Assumption 1: I see that the authors need to contol X_t 2 in Thm 1. (Eq.


Review for NeurIPS paper: Robustness Analysis of Non-Convex Stochastic Gradient Descent using Biased Expectations

Neural Information Processing Systems

After significant discussions with the reviewers, the reviewers were all unanimously in appreciation of the simplicity and cleanliness of the approach presented by the paper. However the authors are strongly encouraged to improve the presentation of the paper - especially the crucial proof of Lemma 1 - multiple steps have been contracted in the presentation and clarifying them is necessary. Furthermore the case of the diminishing step-size scheme is strongly suggested to be fleshed out in theory rather than being left as straightforward extensions. Lastly, the reviewers suggested to use heavier tailed distribution like the Levy distribution to verify the theory better.


Review for NeurIPS paper: Fast and Accurate k -means++ via Rejection Sampling

Neural Information Processing Systems

Additional Feedback: Overall: Why only 3 trees are sufficient for Lemma 3.1? Three looks like a magic number after reading the paper. L90-92 you explain the known results that a single tree metric does not suffice, but why three trees? What are the space requirements of the proposed algorithm? L36-41: In your main contribution, you should *not* Use \tilde{O} without defining explicitly the hidden terms.


Review for NeurIPS paper: Fast and Accurate k -means++ via Rejection Sampling

Neural Information Processing Systems

The paper presents a new algorithm for speeding up k-means algorithms with rigorous theoretical guarantees. It is quite surprising that they can improve the running time to \tilde{O}(nd n {1 \eps}) when even one round of k-means algorithm takes O(ndk) time. The main shortcoming is the performance gain is only visible for large k. However, I think the large k regime is very interesting and does appear in practice. The authors should add discussion about aspect ratio and the new experiments as pointed out by them in the rebuttal.


Review for NeurIPS paper: Improved Guarantees for k-means++ and k-means++ Parallel

Neural Information Processing Systems

Summary and Contributions: In this submission new bounds on the approximation factor of k-means and k-means are presented. The first theoretical contribution is an upper bound of 5(ln(k) 2) on the approximation factor of k-means, which improved upon the previously known upper bound of 8(ln(k) 2) by a constant factor. Then bicriteria approximations are considered, i.e., one uses k-means and k-means to compute clusterings with k Delta clusters for some Delta 0 and compares the costs of these clusterings with those of an optimal k-clustering. Both for k-means and k-means such bicriteria results are already known but in this submission improved bounds are shown. These improved bounds exhibit almost the same asymptotic behavior with respect to Delta as the known bounds but the constants are significantly smaller.


Review for NeurIPS paper: Improved Guarantees for k-means++ and k-means++ Parallel

Neural Information Processing Systems

Although the improved analysis does not offer a huge jump compared to the known bounds, the importance of this algorithm and the simplification of the proof makes the paper an important contribution. Please give some discussion on how your results are orthogonal to the recent results by Rozhon (we agree that they are orthogonal).


SyMANTIC: An Efficient Symbolic Regression Method for Interpretable and Parsimonious Model Discovery in Science and Beyond

arXiv.org Artificial Intelligence

Symbolic regression (SR) is an emerging branch of machine learning focused on discovering simple and interpretable mathematical expressions from data. Although a wide-variety of SR methods have been developed, they often face challenges such as high computational cost, poor scalability with respect to the number of input dimensions, fragility to noise, and an inability to balance accuracy and complexity. This work introduces SyMANTIC, a novel SR algorithm that addresses these challenges. SyMANTIC efficiently identifies (potentially several) low-dimensional descriptors from a large set of candidates (from $\sim 10^5$ to $\sim 10^{10}$ or more) through a unique combination of mutual information-based feature selection, adaptive feature expansion, and recursively applied $\ell_0$-based sparse regression. In addition, it employs an information-theoretic measure to produce an approximate set of Pareto-optimal equations, each offering the best-found accuracy for a given complexity. Furthermore, our open-source implementation of SyMANTIC, built on the PyTorch ecosystem, facilitates easy installation and GPU acceleration. We demonstrate the effectiveness of SyMANTIC across a range of problems, including synthetic examples, scientific benchmarks, real-world material property predictions, and chaotic dynamical system identification from small datasets. Extensive comparisons show that SyMANTIC uncovers similar or more accurate models at a fraction of the cost of existing SR methods.


Multi-level Supervised Contrastive Learning

arXiv.org Artificial Intelligence

Contrastive learning is a well-established paradigm in representation learning. The standard framework of contrastive learning minimizes the distance between "similar" instances and maximizes the distance between dissimilar ones in the projection space, disregarding the various aspects of similarity that can exist between two samples. Current methods rely on a single projection head, which fails to capture the full complexity of different aspects of a sample, leading to suboptimal performance, especially in scenarios with limited training data. In this paper, we present a novel supervised contrastive learning method in a unified framework called multilevel contrastive learning (MLCL), that can be applied to both multi-label and hierarchical classification tasks. The key strength of the proposed method is the ability to capture similarities between samples across different labels and/or hierarchies using multiple projection heads. Extensive experiments on text and image datasets demonstrate that the proposed approach outperforms state-of-the-art contrastive learning methods