infact
Statistically Valid Hyperparameter Selection: From Tuning to Guarantees
Farzaneh, Amirmohammad, Simeone, Osvaldo
Hyperparameter selection is a critical step in the deployment of modern artificial intelligence systems, given the need to tune degrees of freedom such as inference-time parameters, implementation-level settings, and thresholds driving decision rules. Despite its practical importance, hyperparameter selection is typically performed using best-effort empirical methods such as grid search or Bayesian optimization, which provide no formal statistical guarantees on reliability or safety. This monograph presents a unified statistical framework for reliable hyperparameter selection, centered on the learn-then-test (LTT) paradigm, which formulates the problem as multiple hypothesis testing over a candidate set of hyperparameters. The framework enables the selection of hyperparameters that provably satisfy application-specific reliability requirements -- such as bounds on average risk, quantile risk, or information-theoretic constraints -- with explicit, finite-sample control of error probabilities. The supporting statistical machinery, namely p-values, e-values, and concentration inequalities, is developed from first principles in a dedicated appendix.
7716d0fc31636914783865d34f6cdfd5-AuthorFeedback.pdf
This is becausea>t a takes a large amount of iterations to increase from negative to0.26 Consequently,withalargestepsize,wcanmovefarawayfromw beforea>t a becomesnonnegative. For problems with multiple global optima, our analysis can still be applied if the35 following condition holds: there exists one global optimum such that the PD condition holds globally with respect36 tothis optimum.
48aedb8880cab8c45637abc7493ecddd-AuthorFeedback.pdf
Infact,ourexperiments are35 designed todemonstrate thatthevGraph frameworkenables community detection andnode representation learning36 to benefit one other, not to prove that it outperforms all existing studies. Therefore, we decided to choose certain37 representativemethods(i.e.,matrixfactorization-based methods,generativemodels,andK-Meansafternodeembed-38 dings) which help validate this point. We will discuss more studies in the revised draft.(2)AboutchoosingK. In39 practice, when the trueK is not given, we can still chooseK according to the performance on validation set (as in40 [14,36]).
AUnifyingPost-Processing Frameworkfor Multi-ObjectiveLearn-to-DeferProblems
Inthisparadigm, wepermit thesystem to defer a subset of its tasks to the expert. Although there are currently systems that follow this paradigm and are designed to optimize the accuracy of the final human-AI team, the general methodology for developing such systems under a set of constraints (e.g., algorithmic fairness, expert intervention budget, defer of anomaly,etc.)