Neural system identification for large populations separating “what” and “where”

Neuroscientists classify neurons into different types tha t perform similar computations at different locations in the visual field. Traditio nal methods for neural system identification do not capitalize on this separation o f "what" and "where". Learning deep convolutional feature spaces that are shared among many neurons provides an exciting path forward, but the architectural de sign needs to account for data limitations: While new experimental techniques enabl e recordings from thousands of neurons, experimental time is limited so that one ca n sample only a small fraction of each neuron's response space. Here, we show that a major bottleneck for fitting convolutional neural networks (CNNs) to neural d ata is the estimation of the individual receptive field locations - a problem that h as been scratched only at the surface thus far. W e propose a CNN architecture with a s parse readout layer factorizing the spatial (where) and feature (what) dimensi ons. Our network scales well to thousands of neurons and short recordings and can be t rained end-to-end. W e evaluate this architecture on ground-truth data to explo re the challenges and limitations of CNN-based system identification. Moreover, we show that our network model outperforms current state-of-the art system ide ntification models of mouse primary visual cortex.