Current work on human-machine alignment aims at understanding machine-learned latent spaces and their correspondence to human representations. G{\"a}rdenfors' conceptual spaces is a prominent framework for understanding human representations. Convexity of object regions in conceptual spaces is argued to promote generalizability, few-shot learning, and interpersonal alignment. Based on these insights, we investigate the notion of convexity of concept regions in machine-learned latent spaces. We develop a set of tools for measuring convexity in sampled data and evaluate emergent convexity in layered representations of state-of-the-art deep networks. We show that convexity is robust to basic re-parametrization and, hence, meaningful as a quality of machine-learned latent spaces. We find that approximate convexity is pervasive in neural representations in multiple application domains, including models of images, audio, human activity, text, and medical images. Generally, we observe that fine-tuning increases the convexity of label regions. We find evidence that pretraining convexity of class label regions predicts subsequent fine-tuning performance.

Probabilistic machine learning provides a framework in which it is possible to reason about uncertainty for both models and predictions (Ghahramani, 2015). It is often argued that especially in high-stakes applications (healthcare, robotics, etc.), uncertainty estimates for decisions/predictions should be a central component and that they should be well-calibrated (Kuleshov and Deshpande, 2022). The intuition behind calibration is that the uncertainty estimates should accurately reflect the reality; for example if a classification model predicts 80% probability of belonging to class A on 10 datapoints, then (on average) we would expect 8 of those 10 samples actually belong to class A. Likewise - but less intuitively - in regression, if a calibrated model generates a prediction µ and uncertainty estimate σ, we would see p percent of the data lying inside a p percentile confidence interval of µ (Busk et al., 2021). In general, uncertainty can be divided into aleatoric (irreducible inherent randomness in the data-generating process) and epistemic (lack of knowledge, i.e. it can be reduced if more data is collected) (Hüllermeier and Waegeman, 2021). However, this distinction is rarely used when evaluating uncertainty estimates for regression tasks and although this has been critiqued (Sluijterman et al., 2021), it is highly non-trivial to achieve for real-world applications because it normally requires access to the underlying true function. Uncertainty also plays a central role Bayesian Optimization (BO) (Snoek et al., 2012), which will be the focus of this paper. As a sequential design strategy for global optimization, BO has several applications with perhaps the most popular ones being general experimental design (Shahriari et al., 2015) and model selection for machine learning