Searching in Space and Time: Unified Memory-Action Loops for Open-World Object Retrieval