Goto

Collaborating Authors

 infact


Statistically Valid Hyperparameter Selection: From Tuning to Guarantees

arXiv.org Machine Learning

Hyperparameter selection is a critical step in the deployment of modern artificial intelligence systems, given the need to tune degrees of freedom such as inference-time parameters, implementation-level settings, and thresholds driving decision rules. Despite its practical importance, hyperparameter selection is typically performed using best-effort empirical methods such as grid search or Bayesian optimization, which provide no formal statistical guarantees on reliability or safety. This monograph presents a unified statistical framework for reliable hyperparameter selection, centered on the learn-then-test (LTT) paradigm, which formulates the problem as multiple hypothesis testing over a candidate set of hyperparameters. The framework enables the selection of hyperparameters that provably satisfy application-specific reliability requirements -- such as bounds on average risk, quantile risk, or information-theoretic constraints -- with explicit, finite-sample control of error probabilities. The supporting statistical machinery, namely p-values, e-values, and concentration inequalities, is developed from first principles in a dedicated appendix.


Oracle-EfficientAlgorithmsfor OnlineLinearOptimizationwithBanditFeedback

Neural Information Processing Systems

We propose computationally efficient algorithms foronline linear optimization with bandit feedback, in which a player chooses anaction vectorfrom a given (possibly infinite) setA Rd, and then suffers a loss that can be expressed as a linear function in action vectors.




7716d0fc31636914783865d34f6cdfd5-AuthorFeedback.pdf

Neural Information Processing Systems

This is becausea>t a takes a large amount of iterations to increase from negative to0.26 Consequently,withalargestepsize,wcanmovefarawayfromw beforea>t a becomesnonnegative. For problems with multiple global optima, our analysis can still be applied if the35 following condition holds: there exists one global optimum such that the PD condition holds globally with respect36 tothis optimum.


48aedb8880cab8c45637abc7493ecddd-AuthorFeedback.pdf

Neural Information Processing Systems

Infact,ourexperiments are35 designed todemonstrate thatthevGraph frameworkenables community detection andnode representation learning36 to benefit one other, not to prove that it outperforms all existing studies. Therefore, we decided to choose certain37 representativemethods(i.e.,matrixfactorization-based methods,generativemodels,andK-Meansafternodeembed-38 dings) which help validate this point. We will discuss more studies in the revised draft.(2)AboutchoosingK. In39 practice, when the trueK is not given, we can still chooseK according to the performance on validation set (as in40 [14,36]).



AUnifyingPost-Processing Frameworkfor Multi-ObjectiveLearn-to-DeferProblems

Neural Information Processing Systems

Inthisparadigm, wepermit thesystem to defer a subset of its tasks to the expert. Although there are currently systems that follow this paradigm and are designed to optimize the accuracy of the final human-AI team, the general methodology for developing such systems under a set of constraints (e.g., algorithmic fairness, expert intervention budget, defer of anomaly,etc.)


ACentralLimitTheoremforDifferentiallyPrivate QueryAnswering

Neural Information Processing Systems

The central question is,therefore, tounderstand which noise distribution optimizes the privacy-accuracy trade-off, especially when the dimension of the answer vector ishigh.