Greedy inference with structure-exploiting lazy maps