PALMER: Perception - Action Loop with Memory for Long-Horizon Planning