Sketch Recognition with Natural Correction and Editing

AAAI Conferences

In this paper, we target at the problem of sketch recognition. We systematically study how to incorporate users' correction and editing into isolated and full sketch recognition. This is a natural and necessary interaction in real systems such as Visio where very similar shapes exist. First, a novel algorithm is proposed to mine the prior shape knowledge for three editing modes. Second, to differentiate visually similar shapes, a novel symbol recognition algorithm is introduced by leveraging the learnt shape knowledge. Then, a novel editing detection algorithm is proposed to facilitate symbol recognition. Furthermore, both of the symbol recognizer and the editing detector are systematically incorporated into the full sketch recognition. Finally, based on the proposed algorithms, a real-time sketch recognition system is built to recognize hand-drawn flowcharts and diagrams with flexible interactions. Extensive experiments show the effectiveness of the proposed algorithms.


Grouping Strokes into Shapes in Hand-Drawn Diagrams

AAAI Conferences

Objects in freely-drawn sketches often have no spatial or temporal separation, making object recognition difficult. We present a two-step stroke-grouping algorithm that first classifies individual strokes according to the type of object to which they belong, then groups strokes with like classifications into clusters representing individual objects. The first step facilitates clustering by naturally separating the strokes, and both steps fluidly integrate spatial and temporal information. Our approach to grouping is unique in its formulation as an efficient classification task rather than, for example, an expensive search task. Our single-stroke classifier performs at least as well as existing single-stroke classifiers on text vs. nontext classification, and we present the first three-way single-stroke classification results. Our stroke grouping results are the first reported of their kind; our grouping algorithm correctly groups between 86% and 91% of the ink in diagrams from two domains, with between 69% and 79% of shapes being perfectly clustered.


Learning from Neighboring Strokes: Combining Appearance and Context for Multi-Domain Sketch Recognition

Neural Information Processing Systems

We propose a new sketch recognition framework that combines a rich representation of low level visual appearance with a graphical model for capturing high level relationships between symbols. This joint model of appearance and context allows our framework to be less sensitive to noise and drawing variations, improving accuracy and robustness. The result is a recognizer that is better able to handle the wide range of drawing styles found in messy freehand sketches. We evaluate our work on two real-world domains, molecular diagrams and electrical circuit diagrams, and show that our combined approach significantly improves recognition performance.


Combining Geometry and Domain Knowledge to Interpret Hand-Drawn Diagrams

AAAI Conferences

We present a sketch understanding system for networklike diagrams consisting of symbols linked together. This system employs a novel parser to automatically extract symbols from a continuous stream of pen strokes. The parser uses geometric information to enumerate candidate symbols, and then uses domain knowledge to prune away unlikely candidates. The candidates are classified with a novel, domainindependent, probabilistic, feature-based symbol recognizer. Domain knowledge and context are used to correct parsing and recognition errors. To demonstrate our system, we used it to create a sketch-based interface for an electric circuit analysis program.


Recognition of Hand Drawn Chemical Diagrams

AAAI Conferences

Chemists often use hand-drawn structural diagrams to capture and communicate ideas about organic compounds. However, the software available today for specifying these structures to a computer relies on a traditional mouse and keyboard interface, and as a result lacks the ease of use, naturalness, and speed of drawing on paper. In response, we have developed a novel sketch-based system capable of interpreting handdrawn organic chemistry diagrams, allowing users to draw molecules with a pen-based input device in much the same way that they would on paper. The system's ability to interpret a sketch is based on knowledge about both chemistry and chemical drawing conventions. The system employs a trainable symbol recognizer incorporating both feature-based and image-based methods to locate and identify symbols in the sketch. Analysis of the spatial context around each symbol allows the system to choose among competing interpretations and determine an initial structure for the molecule. Finally, knowledge of chemistry (in particular chemical valence) enables the system to check the validity of its interpretation and, when necessary, refine it to recover from inconsistencies. We demonstrate that the system is capable of recognizing diagrams of common organic molecules and show that using domain knowledge produces a noticeable improvement in recognition accuracy.