Reviews: Multi-objective Bayesian optimisation with preferences over objectives

Summary: This paper proposes a method for multi-objective Bayesian optimization when a user has given "preference order constraints", i.e. preferences about the importance of different objectives. For example, a user might specify that he or she wants to determine where, along the pareto front, a given objective varies significantly with respect to other objectives (which the authors term "diversity") or when the objective is static with respect to other objectives (which they term "stability"). The authors give algorithms for this setting and show empirical results on synthetic functions and on a model search task. Comments: My main criticism of this paper is that I am not convinced about the motivation for, and uses cases of, the described task of finding regions of the pareto front where an objective is "diverse" or "stable" as they are defined in the paper. There are two potential examples given in the introduction, but these are brief and unconvincing (another comment on these below). A real experiment is shown on a neural network model search task, but it is unclear how the method, when applied here, provides real benefits over other multi-objective optimization methods.