Deliberate Planning in Language Models with Symbolic Representation