FastMem: Fast Memorization of Prompt Improves Context Awareness of Large Language Models