Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models