Inverting Deep Generative models, One layer at a time