Unlocking Anticipatory Text Generation: A Constrained Approach for Faithful Decoding with Large Language Models