Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

Table 1 summarises various approaches for learning to defer.

We thank all the reviewers ( R1, R2, R3) for their feedback and suggestions. However, this is not paired with an increase in classification accuracy. We will add the suggested experiment to Section 3.3. On CIFAR-100, we obtain an accuracy of 55.5 The proposed loss requires a similarity to only one class during training. 's suggestion, we have quantified the relation The results are shown in Table B. Our proposal We will add the discussions to Section 3.1.

Previous work has proposed many new loss functions and regularizers that improve test accuracy on image classification tasks. However, it is not clear whether these loss functions learn better representations for downstream tasks. This paper studies how the choice of training objective affects the transferability of the hidden representations of convolutional neural networks trained on ImageNet. We show that many objectives lead to statistically significant improvements in ImageNet accuracy over vanilla softmax cross-entropy, but the resulting fixed feature extractors transfer substantially worse to downstream tasks, and the choice of loss has little effect when networks are fully fine-tuned on the new tasks. Using centered kernel alignment to measure similarity between hidden representations of networks, we find that differences among loss functions are apparent only in the last few layers of the network. We delve deeper into representations of the penultimate layer, finding that different objectives and hyperparameter combinations lead to dramatically different levels of class separation. Representations with higher class separation obtain higher accuracy on the original task, but their features are less useful for downstream tasks. Our results suggest there exists a trade-off between learning invariant features for the original task and features relevant for transfer tasks.