Similarity of Neural Network Models: A Survey of Functional and Representational Measures

However, understanding and measuring similarity of neural networks is a complex problem, as there are multiple perspectives on how such models can be similar. In this work, we specifically focus on two key perspectives: representational and functional measures of similarity (see Figure 1). Representational similarity measures assess how activations of intermediate layers differ, whereas functional similarity measures specifically compare the outputs of neural networks with respect to their task. Both perspectives on their own are not sufficient to gain detailed insights into similarity of neural network models. Seemingly similar representations can still yield different outputs, and conversely, similar outputs can result from different representations. In that sense, combining these two complementary perspectives provides a more comprehensive approach to analyze similarity between neural networks at all layers. Given the broad range of research on neural network similarity, numerous similarity measures have been proposed and applied, often with lines of research being disconnected from each other. With this work, we provide a comprehensive overview of measures for representational similarity and functional similarity that gives a unified perspective on the existing literature and can inform and guide both researchers and practitioners interested in understanding and comparing neural network models.