How can we build AI systems that can learn any set of individual human values both quickly and safely, avoiding causing harm or violating societal standards for acceptable behavior during the learning process? We explore the effects of representational alignment between humans and AI agents on learning human values.

We first demonstrate theoretically that off-manifold robustness leads input gradients to lie approximately on the data manifold, explaining their perceptual alignment. We then show that Bayes optimal models satisfy off-manifold robustness, and confirm the same empirically for robust models trained via gradient norm regularization, randomized smoothing, and adversarial training with projected gradient descent. Quantifying the perceptual alignment of model gradients via their similarity with the gradients of generative models, we show that off-manifold robustness correlates well with perceptual alignment. Finally, based on the levels of on-and off-manifold robustness, we identify three different regimes of robustness that affect both perceptual alignment and model accuracy: weak robustness, bayes-aligned robustness, and excessive robustness.

Optimization algorithms on Riemannian manifolds used in this paper to optimize the acquisition function inageometry-awaremanner,havebeen developed bytaking advantage oftheEuclidean tangent spaceTxMlinked to each pointxon the manifoldM.

Foreacharma, letr(a) and cj(a) be, resp., the meanrewardandmeanresource-j consumption,i.e.,(r(a);c1(a),..., cd(a)):=Eo Da[o].We sometimeswriter =( r(a): a 2 [K])andcj =( cj(a): a 2 [K])asvectorsoverarms. Second, weuseatighterversionof Eq. (3.6) (see AppendixD.3):