Language Model as Planner and Formalizer under Constraints