Figure A.1: Among all hyperparameter combinations considered in our grid search, a combination of ( We used a compute time of approximately 5600 CPU-hours of 2.90GHz Intel Xeon Gold In this section, we outline our anomaly detection experimental setting in more detail. Given a dataset (e.g., CIFAR-10) with In contrast to the "one-vs-rest" setting, in LOO we define one class of the In both "one-vs-rest" and LOO AD settings, we evaluate model representations in the following way: We show the pairs of items that change the most in distance in Table B.1. "stethoscope", which are semantically unrelated but perhaps have some slight visual similarity, tend We show the results in Fig. B.1. Table B.1: Distances between pairs of individual items from THINGS, ranked by the relative change in cosine The top items move much closer together under naive alignment, while the bottom ones move much farther apart. Figure B.1: How does the global structure of the representations change after alignment?

Representation learning usually involves pretraining a network on a large, diverse dataset using one of a handful of different types of supervision.

Figure A.1: Among all hyperparameter combinations considered in our grid search, a combination of ( We used a compute time of approximately 5600 CPU-hours of 2.90GHz Intel Xeon Gold In this section, we outline our anomaly detection experimental setting in more detail. Given a dataset (e.g., CIFAR-10) with In contrast to the "one-vs-rest" setting, in LOO we define one class of the In both "one-vs-rest" and LOO AD settings, we evaluate model representations in the following way: We show the pairs of items that change the most in distance in Table B.1. "stethoscope", which are semantically unrelated but perhaps have some slight visual similarity, tend We show the results in Fig. B.1. Table B.1: Distances between pairs of individual items from THINGS, ranked by the relative change in cosine The top items move much closer together under naive alignment, while the bottom ones move much farther apart. Figure B.1: How does the global structure of the representations change after alignment?

Deep neural networks have reached human-level performance on many computer vision tasks. However, the objectives used to train these networks enforce only that similar images are embedded at similar locations in the representation space, and do not directly constrain the global structure of the resulting space. Here, we explore the impact of supervising this global structure by linearly aligning it with human similarity judgments. We find that a naive approach leads to large changes in local representational structure that harm downstream performance. Thus, we propose a novel method that aligns the global structure of representations while preserving their local structure. This global-local transform considerably improves accuracy across a variety of few-shot learning and anomaly detection tasks. Our results indicate that human visual representations are globally organized in a way that facilitates learning from few examples, and incorporating this global structure into neural network representations improves performance on downstream tasks.