Lee, Jimin
SAFE-SQL: Self-Augmented In-Context Learning with Fine-grained Example Selection for Text-to-SQL
Lee, Jimin, Baek, Ingeol, Kim, Byeongjeong, Lee, Hwanhee
Text-to-SQL aims to convert natural language questions into executable SQL queries. While previous approaches, such as skeleton-masked selection, have demonstrated strong performance by retrieving similar training examples to guide large language models (LLMs), they struggle in real-world scenarios where such examples are unavailable. To overcome this limitation, we propose Self-Augmentation in-context learning with Fine-grained Example selection for Text-to-SQL (SAFE-SQL), a novel framework that improves SQL generation by generating and filtering self-augmented examples. SAFE-SQL first prompts an LLM to generate multiple Text-to-SQL examples relevant to the test input. Then SAFE-SQL filters these examples through three relevance assessments, constructing high-quality in-context learning examples. Using self-generated examples, SAFE-SQL surpasses the previous zero-shot, and few-shot Text-to-SQL frameworks, achieving higher execution accuracy. Notably, our approach provides additional performance gains in extra hard and unseen scenarios, where conventional methods often fail.
Interactive Sketchpad: An Interactive Multimodal System for Collaborative, Visual Problem-Solving
Chen, Steven-Shine, Lee, Jimin, Liang, Paul Pu
Humans have long relied on visual aids like sketches and diagrams to support reasoning and problem-solving. Visual tools, like auxiliary lines in geometry or graphs in calculus, are essential for understanding complex ideas. However, many tutoring systems remain text-based, providing feedback only through natural language. Leveraging recent advances in Large Multimodal Models (LMMs), this paper introduces Interactive Sketchpad, a tutoring system that combines language-based explanations with interactive visualizations to enhance learning. Built on a pre-trained LMM, Interactive Sketchpad is fine-tuned to provide step-by-step guidance in both text and visuals, enabling natural multimodal interaction with the student. Accurate and robust diagrams are generated by incorporating code execution into the reasoning process. User studies conducted on math problems such as geometry, calculus, and trigonometry demonstrate that Interactive Sketchpad leads to improved task comprehension, problem-solving accuracy, and engagement levels, highlighting its potential for transforming educational technologies.
Probing-RAG: Self-Probing to Guide Language Models in Selective Document Retrieval
Baek, Ingeol, Chang, Hwan, Kim, Byeongjeong, Lee, Jimin, Lee, Hwanhee
Retrieval-Augmented Generation (RAG) enhances language models by retrieving and incorporating relevant external knowledge. However, traditional retrieve-and-generate processes may not be optimized for real-world scenarios, where queries might require multiple retrieval steps or none at all. In this paper, we propose a Probing-RAG, which utilizes the hidden state representations from the intermediate layers of language models to adaptively determine the necessity of additional retrievals for a given query. By employing a pre-trained prober, Probing-RAG effectively captures the model's internal cognition, enabling reliable decision-making about retrieving external documents. Experimental results across five open-domain QA datasets demonstrate that Probing-RAG outperforms previous methods while reducing the number of redundant retrieval steps.
Crafting the Path: Robust Query Rewriting for Information Retrieval
Baek, Ingeol, Lee, Jimin, Yang, Joonho, Lee, Hwanhee
Query rewriting aims to generate a new query that can complement the original query to improve the information retrieval system. Recent studies on query rewriting, such as query2doc (Q2D), query2expand (Q2E) and querey2cot (Q2C), rely on the internal knowledge of Large Language Models (LLMs) to generate a relevant passage to add information to the query. Nevertheless, the efficacy of these methodologies may markedly decline in instances where the requisite knowledge is not encapsulated within the model's intrinsic parameters. In this paper, we propose a novel structured query rewriting method called Crafting the Path tailored for retrieval systems. Crafting the Path involves a three-step process that crafts query-related information necessary for finding the passages to be searched in each step. Specifically, the Crafting the Path begins with Query Concept Comprehension, proceeds to Query Type Identification, and finally conducts Expected Answer Extraction. Experimental results show that our method outperforms previous rewriting methods, especially in less familiar domains for LLMs. We demonstrate that our method is less dependent on the internal parameter knowledge of the model and generates queries with fewer factual inaccuracies. Furthermore, we observe that Crafting the Path has less latency compared to the baselines.
Intelligent upper-limb exoskeleton integrated with soft wearable bioelectronics and deep-learning for human intention-driven strength augmentation based on sensory feedback
Lee, Jinwoo, Kwon, Kangkyu, Soltis, Ira, Matthews, Jared, Lee, Yoonjae, Kim, Hojoong, Romero, Lissette, Zavanelli, Nathan, Kwon, Youngjin, Kwon, Shinjae, Lee, Jimin, Na, Yewon, Lee, Sung Hoon, Yu, Ki Jun, Shinohara, Minoru, Hammond, Frank L., Yeo, Woon-Hong
ABSTRACT The age and stroke-associated decline in musculoskeletal strength degrades the ability to perform daily human tasks using the upper extremities. Here, we introduce an intelligent upper-limb exoskeleton system that uses cloud-based deep learning to predict human intention for strength augmentation. The embedded soft wearable sensors provide sensory feedback by collecting real-time muscle signals, which are simultaneously computed to determine the user's intended movement. The cloud-based deep-learning predicts four upper-limb joint motions with an average accuracy of 96.2% at a 500-550 millisecond response rate, suggesting that the exoskeleton operates just by human intention. In addition, an array of soft pneumatics assists the intended movements by providing 897 newton of force while generating displacement of 87 millimeter at maximum. Collectively, the intent-driven exoskeleton can reduce human muscle activities by 3.7 times on average compared to the unassisted exoskeleton. INTRODUCTION Many individuals suffer from neuromotor disorders that primarily arise from stroke-induced and age-associated declines in musculoskeletal strength and control. Statistically, strokes affect one out of four adults over the age of 25 in their lifetime, and 12.2 million of the global population experience stroke each year Such a disorder restricts the functional independence of the inflicted population because the reduced motor control and unwanted tremor of the upper limb usually pose considerable difficulties in performing everyday tasks that require the dexterity of the upper limbs. Moreover, neuromotor disorders generate tremendous social expenditure in healthcare. However, the previously reported exoskeletons cannot provide pragmatic solutions because they lack essential functionalities to augment the upper-extremity movements. Another limitation of the previously reported exoskeletons is their structural design. In addition, sensory haptic feedback in human assistive robotics is crucial because it translates human physiological signals into strength augmentation. In this context, electromyography (EMG) signals can offer direct information about upper-extremity movements as EMG records the electrical signals in the presence of muscle activities.