On Memory Construction and Retrieval for Personalized Conversational Agents