SEM: Enhancing Spatial Understanding for Robust Robot Manipulation