The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement