Memory-Enhanced Neural Solvers for Efficient Adaptation in Combinatorial Optimization