The Importance of Directional Feedback for LLM-based Optimizers