Statistical Learning
NICE^k Metrics: Unified and Multidimensional Framework for Evaluating Deterministic Solar Forecasting Accuracy
Voyant, Cyril, Despotovic, Milan, Garcia-Gutierrez, Luis, Silva, Rodrigo Amaro e, Lauret, Philippe, Soubdhan, Ted, Bailek, Nadjem
Accurate solar energy output prediction is key for integrating renewables into grids, maintaining stability, and improving energy management. However, standard error metrics such as Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and Skill Scores (SS) fail to capture the multidimensional nature of solar irradiance forecasting. These metrics lack sensitivity to forecastability, rely on arbitrary baselines (e.g., clear-sky models), and are poorly suited for operational use. To address this, we introduce the NICEk framework (Normalized Informed Comparison of Errors, with k = 1, 2, 3, Sigma), offering a robust and interpretable evaluation of forecasting models. Each NICEk score corresponds to an Lk norm: NICE1 targets average errors, NICE2 emphasizes large deviations, NICE3 highlights outliers, and NICESigma combines all. Using Monte Carlo simulations and data from 68 stations in the Spanish SIAR network, we evaluated methods including autoregressive models, extreme learning, and smart persistence. Theoretical and empirical results align when assumptions hold (e.g., R^2 ~ 1.0 for NICE2). Most importantly, NICESigma consistently shows higher discriminative power (p < 0.05), outperforming traditional metrics (p > 0.05). The NICEk metrics exhibit stronger statistical significance (e.g., p-values from 10^-6 to 0.004 across horizons) and greater generalizability. They offer a unified and operational alternative to standard error metrics in deterministic solar forecasting.
Structure Maintained Representation Learning Neural Network for Causal Inference
Sun, Yang, Lu, Wenbin, Zhou, Yi-Hui
Recent developments in causal inference have greatly shifted the interest from estimating the average treatment effect to the individual treatment effect. In this article, we improve the predictive accuracy of representation learning and adversarial networks in estimating individual treatment effects by introducing a structure keeper which maintains the correlation between the baseline covariates and their corresponding representations in the high dimensional space. We train a discriminator at the end of representation layers to trade off representation balance and information loss. We show that the proposed discriminator minimizes an upper bound of the treatment estimation error. We can address the tradeoff between distribution balance and information loss by considering the correlations between the learned representation space and the original covariate feature space. We conduct extensive experiments with simulated and real-world observational data to show that our proposed Structure Maintained Representation Learning (SMRL) algorithm outperforms state-of-the-art methods. We also demonstrate the algorithms on real electronic health record data from the MIMIC-III database.
Multi-Community Spectral Clustering for Geometric Graphs
Allem, Luiz Emilio, Avrachenkov, Konstantin, Hoppen, Carlos, Manjunath, Hariprasad, Sibemberg, Lucas Siviero
In this paper, we consider the soft geometric block model (SGBM) with a fixed number $k \geq 2$ of homogeneous communities in the dense regime, and we introduce a spectral clustering algorithm for community recovery on graphs generated by this model. Given such a graph, the algorithm produces an embedding into $\mathbb{R}^{k-1}$ using the eigenvectors associated with the $k-1$ eigenvalues of the adjacency matrix of the graph that are closest to a value determined by the parameters of the model. It then applies $k$-means clustering to the embedding. We prove weak consistency and show that a simple local refinement step ensures strong consistency. A key ingredient is an application of a non-standard version of Davis-Kahan theorem to control eigenspace perturbations when eigenvalues are not simple. We also analyze the limiting spectrum of the adjacency matrix, using a combination of combinatorial and matrix techniques.
Model Recycling Framework for Multi-Source Data-Free Supervised Transfer Learning
This situation can give rise to privacy concerns, as organizations may not want to share sensitive information; for instance, healthcare providers may be reluctant to share patient information and security system maintainers may not want to risk sharing facial recognition data for system performance updates. Additionally, there may be issues with obtaining the source data such as when it is hard to retrieve due to technical difficulties or intellectual property restrictions (Li et al., 2020b; Chen et al., 2021; Liang et al., 2020; Ahmed et al., 2021b). Recent advancements in source-free unsupervised domain adaptation (SFUDA) have presented solutions for a scenario where source data is not accessible (Fang et al., 2022). Purposely, SFUDA utilizes pre-trained source models to improve the generalization of a model on an unlabeled target dataset. Our work is similar to other approaches in the field of SFUDA (Li et al., 2020b; Chen et al., 2021; Liang et al., 2020; Ahmed et al., 2021b), in that it addresses the practical scenario where source data is not available during training. Importantly, a crucial aspect is often overlooked by the majority of SFUDA studies. When it is assumed that source data is not accessible, then it cannot be guaranteed that the available source models have been trained on domains related to the target task. And yet, most of the works only have experimented on classic domain adaptation benchmarks, which are somewhat related by design, e.g., Digits-Five (Peng et al., 2019), Office-31 (Saenko et al., 2010), and Office-Home (Venkateswara et al., 2017), i.e,, domains that share the same labels but are dissimilar in feature (and ambient) space. Our approach is unique in that we consider such a source-free supervised transfer learning (SFSTL) setting (Lee et al., 2019), where we do not assume source models are trained on tasks with similar feature spaces or 1
Density estimation with atoms, and functional estimation for mixed discrete-continuous data
Saha, Aytijhya, Ramdas, Aaditya
In classical density (or density-functional) estimation, it is standard to assume that the underlying distribution has a density with respect to the Lebesgue measure. However, when the data distribution is a mixture of continuous and discrete components, the resulting methods are inconsistent in theory and perform poorly in practice. In this paper, we point out that a minor modification of existing methods for nonparametric density (functional) estimation can allow us to fully remove this assumption while retaining nearly identical theoretical guarantees and improved empirical performance. Our approach is very simple: data points that appear exactly once are likely to originate from the continuous component, whereas repeated observations are indicative of the discrete part. Leveraging this observation, we modify existing estimators for a broad class of functionals of the continuous component of the mixture; this modification is a "wrapper" in the sense that the user can use any underlying method of their choice for continuous density functional estimation. Our modifications deliver consistency without requiring knowledge of the discrete support, the mixing proportion, and without imposing additional assumptions beyond those needed in the absence of the discrete part. Thus, various theorems and existing software packages can be made automatically more robust, with absolutely no additional price when the data is not truly mixed.
Kernel-Based Sparse Additive Nonlinear Model Structure Detection through a Linearization Approach
Ebrahimkhani, Sadegh, Lataire, John
The choice of parameterization in Nonlinear (NL) system models greatly affects the quality of the estimated model. Overly complex models can be impractical and hard to interpret, necessitating data-driven methods for simpler and more accurate representations. In this paper, we propose a data-driven approach to simplify a class of continuous-time NL system models using linear approximations around varying operating points. Specifically, for sparse additive NL models, our method identifies the number of NL subterms and their corresponding input spaces. Under small-signal operation, we approximate the unknown NL system as a trajectory-scheduled Linear Parameter-Varying (LPV) system, with LPV coefficients representing the gradient of the NL function and indicating input sensitivity. Using this sensitivity measure, we determine the NL system's structure through LPV model reduction by identifying non-zero LPV coefficients and selecting scheduling parameters. We introduce two sparse estimators within a vector-valued Reproducing Kernel Hilbert Space (RKHS) framework to estimate the LPV coefficients while preserving their structural relationships. The structure of the sparse additive NL model is then determined by detecting non-zero elements in the gradient vector (LPV coefficients) and the Hessian matrix (Jacobian of the LPV coefficients). We propose two computationally tractable RKHS-based estimators for this purpose. The sparsified Hessian matrix reveals the NL model's structure, with numerical simulations confirming the approach's effectiveness.
Canoe Paddling Quality Assessment Using Smart Devices: Preliminary Machine Learning Study
Parab, S., Lamelas, A., Hassan, A., Bhote, P.
Over 22 million Americans participate in paddling-related activities annually, contributing to a global paddlesports market valued at 2.4 billion US dollars in 2020. Despite its popularity, the sport has seen limited integration of machine learning (ML) and remains hindered by the cost of coaching and specialized equipment. This study presents a novel AI-based coaching system that uses ML models trained on motion data and delivers stroke feedback via a large language model (LLM). Participants were recruited through a collaboration with the NYU Concrete Canoe Team. Motion data were collected across two sessions, one with suboptimal form and one with corrected technique, using Apple Watches and smartphones secured in sport straps. The data underwent stroke segmentation and feature extraction. ML models, including Support Vector Classifier, Random Forest, Gradient Boosting, and Extremely Randomized Trees, were trained on both raw and engineered features. A web based interface was developed to visualize stroke quality and deliver LLM-based feedback. Across four participants, eight trials yielded 66 stroke samples. The Extremely Randomized Tree model achieved the highest performance with an F score of 0.9496 under five fold cross validation. The web interface successfully provided both quantitative metrics and qualitative feedback. Sensor placement near the wrists improved data quality. Preliminary results indicate that smartwatches and smartphones can enable low cost, accessible alternatives to traditional paddling instruction. While limited by sample size, the study demonstrates the feasibility of using consumer devices and ML to support stroke refinement and technique improvement.
Toward Using Machine Learning as a Shape Quality Metric for Liver Point Cloud Generation
Nguyen, Khoa Tuan, Oh, Gaeun, Park, Ho-min, Tozzi, Francesca, Willaert, Wouter, Vankerschaver, Joris, Rashidian, Niki, De Neve, Wesley
While 3D medical shape generative models such as diffusion models have shown promise in synthesizing diverse and anatomically plausible structures, the absence of ground truth makes quality evaluation challenging. Existing evaluation metrics commonly measure distributional distances between training and generated sets, while the medical field requires assessing quality at the individual level for each generated shape, which demands labor-intensive expert review. In this paper, we investigate the use of classical machine learning (ML) methods and PointNet as an alternative, interpretable approach for assessing the quality of generated liver shapes. We sample point clouds from the surfaces of the generated liver shapes, extract handcrafted geometric features, and train a group of supervised ML and PointNet models to classify liver shapes as good or bad. These trained models are then used as proxy discriminators to assess the quality of synthetic liver shapes produced by generative models. Our results show that ML-based shape classifiers provide not only interpretable feedback but also complementary insights compared to expert evaluation. This suggests that ML classifiers can serve as lightweight, task-relevant quality metrics in 3D organ shape generation, supporting more transparent and clinically aligned evaluation protocols in medical shape modeling.
Detecting COPD Through Speech Analysis: A Dataset of Danish Speech and Machine Learning Approach
Sankey-Olsen, Cuno, Olesen, Rasmus Hvass, Eberhard, Tobias Oliver, Triantafyllopoulos, Andreas, Schuller, Bjรถrn, Aslan, Ilhan
Chronic Obstructive Pulmonary Disease (COPD) is a serious and debilitating disease affecting millions around the world. Its early detection using non-invasive means could enable preventive interventions that improve quality of life and patient outcomes, with speech recently shown to be a valuable biomarker. Yet, its validity across different linguistic groups remains to be seen. To that end, audio data were collected from 96 Danish participants conducting three speech tasks (reading, coughing, sustained vowels). Half of the participants were diagnosed with different levels of COPD and the other half formed a healthy control group. Subsequently, we investigated different baseline models using openSMILE features and learnt x-vector embeddings. We obtained a best accuracy of 67% using openSMILE features and logistic regression. Our findings support the potential of speech-based analysis as a non-invasive, remote, and scalable screening tool as part of future COPD healthcare solutions.
From Binary to Continuous: Stochastic Re-Weighting for Robust Graph Explanation
Chen, Zhuomin, Ni, Jingchao, Salehi, Hojat Allah, Zheng, Xu, Luo, Dongsheng
Graph Neural Networks (GNNs) have achieved remarkable performance in a wide range of graph-related learning tasks. However, explaining their predictions remains a challenging problem, especially due to the mismatch between the graphs used during training and those encountered during explanation. Most existing methods optimize soft edge masks on weighted graphs to highlight important substructures, but these graphs differ from the unweighted graphs on which GNNs are trained. This distributional shift leads to unreliable gradients and degraded explanation quality, especially when generating small, sparse subgraphs. To address this issue, we propose a novel iterative explanation framework which improves explanation robustness by aligning the model's training data distribution with the weighted graph distribution appeared during explanation. Our method alternates between two phases: explanation subgraph identification and model adaptation. It begins with a relatively large explanation subgraph where soft mask optimization is reliable. Based on this subgraph, we assign importance-aware edge weights to explanatory and non-explanatory edges, and retrain the GNN on these weighted graphs. This process is repeated with progressively smaller subgraphs, forming an iterative refinement procedure. We evaluate our method on multiple benchmark datasets using different GNN backbones and explanation methods. Experimental results show that our method consistently improves explanation quality and can be flexibly integrated with different architectures.