End-to-End Egospheric Spatial Memory