A key challenge in understanding the sensory transformations of the visual system is to obtain a highly predictive model that maps natural images to neural responses. Deep neural networks (DNNs) provide a promising candidate for such a model. However, DNNs require orders of magnitude more training data than neuroscientists can collect because experimental recording time is severely limited. This motivates us to find images that train highly-predictive DNNs with as little training data as possible. We propose high-contrast, binarized versions of natural images---termed gaudy images---to efficiently train DNNs to predict higher-order visual cortical responses. In simulation experiments and analyses of real neural data, we find that training DNNs with gaudy images substantially reduces the number of training images needed to accurately predict responses to natural images. We also find that gaudy images, chosen before training, outperform images chosen during training by active learning algorithms. Thus, gaudy images overemphasize features of natural images that are the most important for efficiently training DNNs. We believe gaudy images will aid in the modeling of visual cortical neurons, potentially opening new scientific questions about visual processing.

A key challenge in understanding the sensory transformations of the visual system is to obtain a highly predictive model that maps natural images to neural responses. Deep neural networks (DNNs) provide a promising candidate for such a model. However, DNNs require orders of magnitude more training data than neuroscientists can collect because experimental recording time is severely limited. This motivates us to find images that train highly-predictive DNNs with as little training data as possible. We propose high-contrast, binarized versions of natural images---termed gaudy images---to efficiently train DNNs to predict higher-order visual cortical responses.

Weaknesses: The main weakness of the work to me was the context in which the authors proposed and evaluated the use of gaudy images. Modeling neural activity using DNNs is of great importance in computational neuroscience. However, the authors motivate this specific method because they claim there is not enough neural data to train complex models on normal natural images, either straight from image to neural activity or from pre-trained DNN activities to neural activity. I find both of these claims misleading: there is quite a bit of work successfully training end-to-end deep networks from images to neural predictions even with limited experimental data. I especially object to the author's explanation that linear mappings are used between pre-trained DNNs and neural activity due to lack of data.

This paper had borderline scores. Overall, I think this paper presents a nice core finding that was sufficiently well validated in the context of simulations. The simulated results are reasonably compelling and relatively thorough analyses were presented. In the discussion with reviewers, there was a reasonable consensus that the author response overemphasized the extent to which the reviewers were hung up on the lack of experimental data. While R3 felt most strongly that this specific paper was not strong enough without further empirical validation, this was clarified to not be a bias against simulation papers generally, but rather the reviewer's opinion that they were not convinced this specific paper's results were strong enough without real data.

A key challenge in understanding the sensory transformations of the visual system is to obtain a highly predictive model that maps natural images to neural responses. Deep neural networks (DNNs) provide a promising candidate for such a model. However, DNNs require orders of magnitude more training data than neuroscientists can collect because experimental recording time is severely limited. This motivates us to find images that train highly-predictive DNNs with as little training data as possible. We propose high-contrast, binarized versions of natural images---termed gaudy images---to efficiently train DNNs to predict higher-order visual cortical responses.