Collaborating Authors

sketch understanding

Google adds Digital Ink Recognition API for touch and stylus input to ML Kit


A month after announcing changes to ML Kit, its toolset for developers to infuse apps with AI, Google today launched the Digital Ink Recognition API on Android and iOS to allow developers to create apps where stylus and touch act as inputs. As the name implies, the API -- which is powered by the same technology underpinning Google's Gboard software keyboard, Quick Draw, and AutoDraw -- looks at a user's strokes on the screen and recognizes what they're writing or drawing. Google says that with the new Digital Ink Recognition API, developers can enable users to input text and figures with a finger and stylus or transcribe handwritten notes to make them searchable. Classifiers parse written text into a string of characters; other classifiers describe shapes such as drawings, sketches, and emojis by the class to which they belong (e.g., circle, square, happy face, and so on). The Digital Ink Recognition API performs processing in near-real-time and on-device, according to Google, with support for over 300 languages and more than 25 writing systems including all major Latin languages, Chinese, Japanese, Korean, Arabic, and Cyrillic.

Learning from Neighboring Strokes: Combining Appearance and Context for Multi-Domain Sketch Recognition

Neural Information Processing Systems

We propose a new sketch recognition framework that combines a rich representation of low level visual appearance with a graphical model for capturing high level relationships between symbols. This joint model of appearance and context allows our framework to be less sensitive to noise and drawing variations, improving accuracy and robustness. The result is a recognizer that is better able to handle the wide range of drawing styles found in messy freehand sketches. We evaluate our work on two real-world domains, molecular diagrams and electrical circuit diagrams, and show that our combined approach significantly improves recognition performance. Papers published at the Neural Information Processing Systems Conference.

Interactive Cognitive Assessment Tools: A Case Study on Digital Pens for the Clinical Assessment of Dementia Artificial Intelligence

Interactive cognitive assessment tools may be valuable for doctors and therapists to reduce costs and improve quality in healthcare systems. Use cases and scenarios include the assessment of dementia. In this paper, we present our approach to the semi-automatic assessment of dementia. We describe a case study with digital pens for the patients including background, problem description and possible solutions. We conclude with lessons learned when implementing digital tests, and a generalisation for use outside the cognitive impairments field.

Microsoft's digital ink tool makes sense of your chickenscratch


During its Build conference today, Microsoft introduced Project Ink Analysis, which does exactly what you'd think: Make sense of digital writing. The toolkit both understands words and provides features typically found in text editors, like alignment and bulleting. While Project Ink Analysis is still in its experimental stages, it could obviously help anyone who habitually writes with styluses on digital platforms. It might not garner deep insights into your personality like IBM Watson, but its simple beautification tools can clean up chickenscratch and even translate from 67 languages. It could be plenty useful for all the Surface Pen users out there who want their scrawling handwriting to look just a bit more professional (and legible).

Sketch Worksheets in STEM Classrooms: Two Deployments

AAAI Conferences

Sketching can be a valuable tool for science education, but it is currently underutilized. Sketch worksheets were developed to help change this, by using AI technology to give students immediate feedback and to give instructors assistance in grading. Sketch worksheets use visual representations automatically computed by CogSketch, which are combined with conceptual information from the OpenCyc ontology. Feedback is provided to students by comparing an instructor’s sketch to a student’s sketch, using the Structure-Mapping Engine. This paper describes our experiences in deploying sketch worksheets in two types of classes: Geoscience and AI. Sketch worksheets for introductory geoscience classes were developed by geoscientists at University of Wisconsin-Madison, authored using CogSketch and used in classes at both Wisconsin and Northwestern University. Sketch worksheets were also developed and deployed for a knowledge representation and reasoning course at Northwestern. Our experience indicates that sketch worksheets can provide helpful on-the-spot feedback to students, and significantly improve grading efficiency, to the point where sketching assignments can be more practical to use broadly in STEM education.

Using Analogy to Cluster Hand-Drawn Sketches for Sketch-Based Educational Software

AI Magazine

Useful feedback makes use of models of domain-specific knowledge, especially models that are commonly held by potential students. To empirically determine what these models are, student data can be clustered to reveal common misconceptions or common problem-solving strategies. This article describes how analogical retrieval and generalization can be used to cluster automatically analyzed handdrawn sketches incorporating both spatial and conceptual information. We use this approach to cluster a corpus of hand-drawn student sketches to discover common answers. Common answer clusters can be used for the design of targeted feedback and for assessment.

Windows 10 digital ink: All the improvements with the Fall Creators Update


Inking and navigating with a digital pen or stylus within Windows 10 will become easier within the Fall Creators Update, for those of you who use a tablet as, you know, a tablet. The improvements include two major elements: navigation, including using the pen or stylus to select and scroll text; and better interpretation of inked words as text, via a more accurate and responsive handwriting panel. Combined, it's a love letter of sorts to Surface and other tablet users who use the pen to input data. It's amazing how well Windows can interpret your chicken-scratch into text that can be edited in Word and elsewhere. General Windows 10 users won't be able to take advantage of the new features until the launch of the Fall Creators Update on Oct. 17.

AI System Scores Better Than 75% Of Americans In Visual Intelligence Test


The computation model was built on CogSketch, a sketch-understanding system developed in Forbus' laboratory at Northwestern University. Sketching is a natural activity that people do while thinking or trying to communicate an idea, especially when spatial content is involved. Sketching is also heavily used in engineering and geoscience. CogSketch is used to model spatial understanding and reasoning, making it suitable for research based on sketches, but also for testing against a standardized visual intelligence test such as the Raven's Progressive Matrices test.

Franz Inc Customer Applications: Artificial Intelligence

AITopics Original Links

Northwestern University Application: CogSketch People sketch to work through ideas and to communicate, especially when dealing with spatial matters. Software that could participate in sketching could revolutionize spatial education, and provide a new kind of instrument for cognitive science research, as well as being an important scientific advance in its own right. The goal of the CogSketch project is to do the research and development needed to create a sketch understanding system that can be used as an instrument for cognitive science research and as a platform for educational software. This system, called CogSketch, is being developed by the Spatial Intelligence and Learning Center (SILC), a National Science Foundation Sciences of Learning Center. The vision is that, in ten years or less, sketch-base educational software can be as widely available to students as graphing calculators are today.

A polynomial-time relaxation of the Gromov-Hausdorff distance Machine Learning

The Gromov-Hausdorff distance provides a metric on the set of isometry classes of compact metric spaces. Unfortunately, computing this metric directly is believed to be computationally intractable. Motivated by applications in shape matching and point-cloud comparison, we study a semidefinite programming relaxation of the Gromov-Hausdorff metric. This relaxation can be computed in polynomial time, and somewhat surprisingly is itself a pseudometric. We describe the induced topology on the set of compact metric spaces. Finally, we demonstrate the numerical performance of various algorithms for computing the relaxed distance and apply these algorithms to several relevant data sets. In particular we propose a greedy algorithm for finding the best correspondence between finite metric spaces that can handle hundreds of points.