Bayesian Optimization (BO) is a widely used method for optimizing expensive black-box functions, relying on probabilistic surrogate models such as Gaussian Processes (GPs). The quality of the surrogate model is crucial for good optimization performance, especially in the few-shot setting where only a small number of batches of points can be evaluated. In this setting, the initialization plays a critical role in shaping the surrogate's predictive quality and guiding subsequent optimization. Despite this, practitioners typically rely on (quasi-)random designs to cover the input space. However, such approaches neglect two key factors: (a) random designs may not be space-filling, and (b) efficient hyperparameter learning during initialization is essential for high-quality prediction, which may conflict with space-filling designs. To address these limitations, we propose Hyperparameter-Informed Predictive Exploration (HIPE), a novel acquisition strategy that balances space-filling exploration with hyperparameter learning using information-theoretic principles. We derive a closed-form expression for HIPE in the GP setting and demonstrate its effectiveness through extensive experiments in active learning and few-shot BO. Our results show that HIPE outperforms standard initialization strategies in terms of predictive accuracy, hyperparameter identification, and optimization performance, particularly in large-batch, few-shot settings relevant to many real-world BO applications.

Bayesian Optimization specifically aims to increase sample efficiency for hard optimization algorithms, and consequently can help achieve better solutions without incurring large societal costs. For instance, as demonstrated in this work, automotive design problems may be solved much faster, reducing the amount of computationally costly simulations and thus the energy footprint during development. At the same time, improved solutions mean that high crash safety can be achieved with lighter cars, resulting in fewer resources required for their production and, importantly, improving fuel economy of the whole vehicle fleet. Increased robustness to noisy observations further helps reduce the resources spent on evaluating regions of the search space that are not promising. Improvements to the optimization performance and practicality of multi-objective Bayesian optimization have the potential to allow decision makers to better understand and make more informed decisions across multiple trade-offs. We expect these directions to be particularly important as Bayesian optimization is increasingly used for applications such as recommender systems [35], where auxiliary goals such as fairness must be accounted for. Of course, at the end of the day, exactly what objectives decision makers choose to optimize, and how they balance those trade-offs (and whether that is done in equitable fashion) is up to the individuals themselves. Such a partitioning allows for efficient piece-wise computation of the hypervolume improvement from a new point f(xi) by computing the volume of the intersection of the region dominated exclusively by the new point with ({f(xi),P,r) (and not dominated by the P) with each hyperrectangle Sk.

Despite the predominant use of first-order methods for training deep learning models, second-order methods, and in particular, natural gradient methods, remain of interest because of their potential for accelerating training through the use of curvature information. Several methods with non-diagonal preconditioning matrices, including KFAC, Shampoo, and K-BFGS, have been proposed and shown to be effective. Based on the so-called tensor normal (TN) distribution, we propose and analyze a brand new approximate natural gradient method, Tensor Normal Training (TNT), which like Shampoo, only requires knowledge of the shape of the training parameters. By approximating the probabilistically based Fisher matrix, as opposed to the empirical Fisher matrix, our method uses the block-wise covariance of the sampling based gradient as the pre-conditioning matrix. Moreover, the assumption that the sampling-based (tensor) gradient follows a TN distribution, ensures that its covariance has a Kronecker separable structure, which leads to a tractable approximation to the Fisher matrix. Consequently, TNT's memory requirements and per-iteration computational costs are only slightly higher than those for first-order methods. In our experiments, TNT exhibited superior optimization performance to state-of-the-art first-order methods, and comparable optimization performance to the state-of-the-art second-order methods KFAC and Shampoo. Moreover, TNT demonstrated its ability to generalize as well as first-order methods, while using fewer epochs.