Sketch Recognition based on Manifold Learning

AAAI Conferences

Current feature-based methods for sketch recognition systems rely on human-selected features. Certain machine learning techniques have been found to be good nonlinear features extractors. In this paper, we apply a manifold learning method, kernel Isomap, with a new algorithm for multi-stroke sketch recognition, which significantly outperforms the standard featurebased techniques.


Using a Geometric-Based Sketch Recognition Approach to Sketch Chinese Radicals

AAAI Conferences

Unlike English, where unfamiliar words can be queried for its meaning by typing out its letters, the analogous operation in Chinese is far from trivial due to the nature of its written language. One approach for querying Chinese characters involve referencing their dictionary component called radicals. This is advantageous since users would not need to know their pronunciation nor their stroke-order, a requirement in other querying approaches. Currently though, sketching a character's radical for querying is an unsupported capability in existing systems. Using the geometric-based LADDER sketching language combined with the Sezgin lowlevel recognizer, we were able to construct an application which can first recognize handwritten sketches of Chinese radical, and then output candidate Chinese characters which contain that radical. Thus, we were able to demonstrate that a geometric-based sketch recognition approach can be used to easily build applications for recognizing symbols related to Chinese characters while having reasonable recognition rates. Unlike current image-based recognition systems, our system also maintains stroke order information of characters. Since stroke order is important in written Chinese, our system can be easily expanded for use in Chinese language education by providing visual feedback to students on correct stroke order.


Offline Sketch Parsing via Shapeness Estimation

AAAI Conferences

In this work, we target at the problem of offline sketch parsing, in which the temporal orders of strokes are unavailable. It is more challenging than most of existing work, which usually leverages the temporal information to reduce the search space. Different from traditional approaches in which thousands of candidate groups are selected for recognition, we propose the idea of shapeness estimation to greatly reduce this number in a very fast way. Based on the observation that most of hand-drawn shapes with well-defined closed boundaries can be clearly differentiated from non-shapes if normalized into a very small size, we propose an efficient shapeness estimation method. A compact feature representation as well as its efficient extraction method is also proposed to speed up this process. Based on the proposed shapeness estimation, we present a three-stage cascade framework for offline sketch parsing. The shapeness estimation technique in this framework greatly reduces the number of false positives, resulting in a 96.2% detection rate with only 32 candidate group proposals, which is two orders of magnitude less than existing methods. Extensive experiments show the superiority of the proposed framework over state-of-the-art works on sketch parsing in both effectiveness and efficiency, even though they leveraged the temporal information of strokes.


User Identification by Means of Sketched Stroke Features

AAAI Conferences

We present preliminary results of using physical features of a user's sketching style, such as pen tilt and pressure, to identify a user from their sketched strokes.


Combining Speech and Sketch to Interpret Unconstrained Descriptions of Mechanical Devices

AAAI Conferences

Mechanical design tools would be considerably more useful if we could interact with them in the way that human designers communicate design ideas to one another, i.e., using crude sketches and informal speech. Those crude sketches frequently contain pen strokes of two different sorts, one type portraying device structure, the other denoting gestures, such as arrows used to indicate motion. We report here on techniques we developed that use information from both sketch and speech to distinguish gesture strokes from non-gestures -- a critical first step in understanding a sketch of a device. We collected and analyzed unconstrained device descriptions, which revealed six common types of gestures. Guided by this knowledge, we developed a classifier that uses both sketch and speech features to distinguish gesture strokes from non-gestures. Experiments with our techniques indicate that the sketch and speech modalities alone produce equivalent classification accuracy, but combining them produces higher accuracy.