Putting People in LLMs' Shoes: Generating Better Answers via Question Rewriter