Sketch-Guided Constrained Decoding for Boosting Blackbox Large Language Models without Logit Access

Constrained decoding, a technique for enforcing constraints on language model outputs, offers a way to control text generation without retraining or architectural modifications. Its application is, however, typically restricted to models that give users access to next-token distributions (usually via softmax logits), which poses a limitation with blackbox large language models (LLMs). This paper introduces sketchguided constrained decoding (SGCD), a novel approach to constrained decoding for blackbox LLMs, which operates without access to the logits of the blackbox LLM. SGCD utilizes a locally hosted auxiliary model to refine the output of an unconstrained blackbox LLM, effectively treating this initial output as a "sketch" for further elaboration. This approach is complementary to traditional logit-based techniques and enables the application of constrained decoding in settings where full model transparency Figure 1: Overview of sketch-guided constrained decoding is unavailable. We demonstrate the efficacy of (SGCD). In the initial sketching phase, a blackbox SGCD through experiments in closed information LLM generates a preliminary "sketch" answer without extraction and constituency parsing, showing applying any constraints. Then, in the constrained how it enhances the utility and flexibility of generation phase, an auxiliary model, the constrained blackbox LLMs for complex NLP tasks.