Humans draw to facilitate reasoning: we draw auxiliary lines when solving geometry problems; we mark and circle when reasoning on maps; we use sketches to amplify our ideas and relieve our limited-capacity working memory. However, such actions are missing in current multimodal language models (LMs). Current chain-of-thought and tool-use paradigms only use text as intermediate reasoning steps. In this work, we introduce Sketchpad, a framework that gives multimodal LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts planning and reasoning according to the visual artifacts it has drawn. Different from prior work, which uses text-to-image models to enable LMs to draw, Sketchpad enables LMs to draw with lines, boxes, marks, etc., which is closer to human sketching and better facilitates reasoning.

Humans draw to facilitate reasoning: we draw auxiliary lines when solving geometry problems; we mark and circle when reasoning on maps; we use sketches to amplify our ideas and relieve our limited-capacity working memory. However, such actions are missing in current multimodal language models (LMs). Current chain-of-thought and tool-use paradigms only use text as intermediate reasoning steps. In this work, we introduce Sketchpad, a framework that gives multimodal LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts planning and reasoning according to the visual artifacts it has drawn.

This summer, a spate of new features are headed to Windows 10 by way of the Anniversary Update, Microsoft's next major revision to the OS. Chief among the additions is Windows Ink, an experience specifically designed for digital pen users. The full Ink experience is still months away--longer, if you wait on the fruits of Microsoft's partnership with Wacom, which will reportedly yield a special Ink pen by the holidays. But thanks to the recent, massive Windows 10 Build 14322 that Microsoft released to its Insider beta testers, we've had a chance to try out several aspects of Windows Ink, including Ink Workspace, Sketchpad, Sticky Notes, and more. Click the new pen icon to launch the Windows Ink Workspace apps. If you haven't actually worked with digital ink before, relax: Windows Ink is an optional way to interact with Windows, in much the same way you can use either voice or keyboard to query Cortana.