Greedy inference with layers of lazy maps