Leveraging Semantic and Geometric Information for Zero-Shot Robot-to-Human Handover