South America
An Open-Source American Sign Language Fingerspell Recognition and Semantic Pose Retrieval Interface
This paper introduces an open-source interface for American Sign Language fingerspell recognition and semantic pose retrieval, aimed to serve as a stepping stone towards more advanced sign language translation systems. Utilizing a combination of convolutional neural networks and pose estimation models, the interface provides two modular components: a recognition module for translating ASL fingerspelling into spoken English and a production module for converting spoken English into ASL pose sequences. The system is designed to be highly accessible, user-friendly, and capable of functioning in real-time under varying environmental conditions like backgrounds, lighting, skin tones, and hand sizes. We discuss the technical details of the model architecture, application in the wild, as well as potential future enhancements for real-world consumer applications.
Identifying Technical Debt and Its Types Across Diverse Software Projects Issues
Shivashankar, Karthik, Orucevic, Mili, Kruke, Maren Maritsdatter, Martini, Antonio
Technical Debt (TD) identification in software projects issues is crucial for maintaining code quality, reducing long-term maintenance costs, and improving overall project health. This study advances TD classification using transformer-based models, addressing the critical need for accurate and efficient TD identification in large-scale software development. Our methodology employs multiple binary classifiers for TD and its type, combined through ensemble learning, to enhance accuracy and robustness in detecting various forms of TD. We train and evaluate these models on a comprehensive dataset from GitHub Archive Issues (2015-2024), supplemented with industrial data validation. We demonstrate that in-project fine-tuned transformer models significantly outperform task-specific fine-tuned models in TD classification, highlighting the importance of project-specific context in accurate TD identification. Our research also reveals the superiority of specialized binary classifiers over multi-class models for TD and its type identification, enabling more targeted debt resolution strategies. A comparative analysis shows that the smaller DistilRoBERTa model is more effective than larger language models like GPTs for TD classification tasks, especially after fine-tuning, offering insights into efficient model selection for specific TD detection tasks. The study also assesses generalization capabilities using metrics such as MCC, AUC ROC, Recall, and F1 score, focusing on model effectiveness, fine-tuning impact, and relative performance. By validating our approach on out-of-distribution and real-world industrial datasets, we ensure practical applicability, addressing the diverse nature of software projects.
VERA: Validation and Evaluation of Retrieval-Augmented Systems
Ding, Tianyu, Banerjee, Adi, Mombaerts, Laurent, Li, Yunhong, Borogovac, Tarik, Weinstein, Juan Pablo De la Cruz
The increasing use of Retrieval-Augmented Generation (RAG) systems in various applications necessitates stringent protocols to ensure RAG systems accuracy, safety, and alignment with user intentions. In this paper, we introduce VERA (Validation and Evaluation of Retrieval-Augmented Systems), a framework designed to enhance the transparency and reliability of outputs from large language models (LLMs) that utilize retrieved information. VERA improves the way we evaluate RAG systems in two important ways: (1) it introduces a cross-encoder based mechanism that encompasses a set of multidimensional metrics into a single comprehensive ranking score, addressing the challenge of prioritizing individual metrics, and (2) it employs Bootstrap statistics on LLM-based metrics across the document repository to establish confidence bounds, ensuring the repositorys topical coverage and improving the overall reliability of retrieval systems. Through several use cases, we demonstrate how VERA can strengthen decision-making processes and trust in AI applications. Our findings not only contribute to the theoretical understanding of LLM-based RAG evaluation metric but also promote the practical implementation of responsible AI systems, marking a significant advancement in the development of reliable and transparent generative AI technologies.
Bayesian Network Modeling of Causal Influence within Cognitive Domains and Clinical Dementia Severity Ratings for Western and Indian Cohorts
Kumar, Wupadrasta Santosh, Bhutare, Sayali Rajendra, Sinha, Neelam, Issac, Thomas Gregor
This study investigates the causal relationships between Clinical Dementia Ratings (CDR) and its six domain scores across two distinct aging datasets: the Alzheimer's Disease Neuroimaging Initiative (ADNI) and the Longitudinal Aging Study of India (LASI). Using Directed Acyclic Graphs (DAGs) derived from Bayesian network models, we analyze the dependencies among domain scores and their influence on the global CDR. Our approach leverages the PC algorithm to estimate the DAG structures for both datasets, revealing notable differences in causal relationships and edge strengths between the Western and Indian populations. The analysis highlights a stronger dependency of CDR scores on memory functions in both datasets, but with significant variations in edge strengths and node degrees. By contrasting these findings, we aim to elucidate population-specific differences and similarities in dementia progression, providing insights that could inform targeted interventions and improve understanding of dementia across diverse demographic contexts.
Formalization of Operational Domain and Operational Design Domain for Automated Vehicles
Specifying an Operational Design Domain (ODD) is crucial for safeguarding automated vehicle systems against conditions that exceed their capabilities. Yet, prior definitions of ODD have relied on ambiguous and unclear terms, resulting in numerous misunderstandings and misconceptions. This paper introduces a formal approach to clearly define the Operational Domain (OD) and ODD for automated vehicles. Furthermore, the absence of essential terms, such as the OD, has resulted in the creation of numerous terms that have made things more complicated and confusing. This level of complexity is unacceptable when it comes to developing safety-critical systems, where any uncertainty can lead to significant risks. This study addresses these deficiencies by providing a precise mathematical model of OD and clarifying its relationship with other terms. Also, by formalizing these terms, this work establishes a foundation for developing further concepts such as ODD specification and ODD monitoring, which are explained in this paper.
On the Undecidability of Artificial Intelligence Alignment: Machines that Halt
de Melo, Gabriel Adriano, Maximo, Marcos Ricardo Omena De Albuquerque, Soma, Nei Yoshihiro, de Castro, Paulo Andre Lima
The inner alignment problem, which asserts whether an arbitrary artificial intelligence (AI) model satisfices a non-trivial alignment function of its outputs given its inputs, is undecidable. This is rigorously proved by Rice's theorem, which is also equivalent to a reduction to Turing's Halting Problem, whose proof sketch is presented in this work. Nevertheless, there is an enumerable set of provenly aligned AIs that are constructed from a finite set of provenly aligned operations. Therefore, we argue that the alignment should be a guaranteed property from the AI architecture rather than a characteristic imposed post-hoc on an arbitrary AI model. Furthermore, while the outer alignment problem is the definition of a judge function that captures human values and preferences, we propose that such a function must also impose a halting constraint that guarantees that the AI model always reaches a terminal state in finite execution steps. Our work presents examples and models that illustrate this constraint and the intricate challenges involved, advancing a compelling case for adopting an intrinsically hard-aligned approach to AI systems architectures that ensures halting.
Distributed and Secure Kernel-Based Quantum Machine Learning
Swaminathan, Arjhun, Akgรผn, Mete
Quantum computing promises to revolutionize machine learning, offering significant efficiency gains in tasks such as clustering and distance estimation. Additionally, it provides enhanced security through fundamental principles like the measurement postulate and the no-cloning theorem, enabling secure protocols such as quantum teleportation and quantum key distribution. While advancements in secure quantum machine learning are notable, the development of secure and distributed quantum analogues of kernel-based machine learning techniques remains underexplored. In this work, we present a novel approach for securely computing common kernels, including polynomial, radial basis function (RBF), and Laplacian kernels, when data is distributed, using quantum feature maps. Our methodology introduces a robust framework that leverages quantum teleportation to ensure secure and distributed kernel learning. The proposed architecture is validated using IBM's Qiskit Aer Simulator on various public datasets.
Quantifying the Effectiveness of Student Organization Activities using Natural Language Processing
Taruc, Lyberius Ennio F., De La Cruz, Arvin R.
Student extracurricular activities play an important role in enriching the students' educational experiences. With the increasing popularity of Machine Learning and Natural Language Processing, it becomes a logical step that incorporating ML-NLP in improving extracurricular activities is a potential focus of study in Artificial Intelligence (AI). This research study aims to develop a machine learning workflow that will quantify the effectiveness of student-organized activities based on student emotional responses using sentiment analysis. The study uses the Bidirectional Encoder Representations from Transformers (BERT) Large Language Model (LLM) called via the pysentimiento toolkit, as a Transformer pipeline in Hugging Face. A sample data set from Organization C, a Recognized Student Organization (RSO) of a higher educational institute in the Philippines, College X, was used to develop the workflow. The workflow consisted of data preprocessing, key feature selection, LLM feature processing, and score aggregation, resulting in an Event Score for each data set. The results show that the BERT LLM can also be used effectively in analyzing sentiment beyond product reviews and post comments. For the student affairs offices of educational institutions, this study can provide a practical example of how NLP can be applied to real-world scenarios, showcasing the potential impact of data-driven decision making.
HistoGym: A Reinforcement Learning Environment for Histopathological Image Analysis
Liu, Zhi-Bo, Pang, Xiaobo, Wang, Jizhao, Liu, Shuai, Li, Chen
In pathological research, education, and clinical practice, the decision-making process based on pathological images is critically important. This significance extends to digital pathology image analysis: its adequacy is demonstrated by the extensive information contained within tissue structures, which is essential for accurate cancer classification and grading. Additionally, its necessity is highlighted by the inherent requirement for interpretability in the conclusions generated by algorithms. For humans, determining tumor type and grade typically involves multi-scale analysis, which presents a significant challenge for AI algorithms. Traditional patch-based methods are inadequate for modeling such complex structures, as they fail to capture the intricate, multi-scale information inherent in whole slide images. Consequently, there is a pressing need for advanced AI techniques capable of efficiently and accurately replicating this complex analytical process. To address this issue, we introduce HistoGym, an open-source reinforcement learning environment for histopathological image analysis. Following OpenAI Gym APIs, HistoGym aims to foster whole slide image diagnosis by mimicking the real-life processes of doctors. Leveraging the pyramid feature of WSIs and the OpenSlide API, HistoGym provides a unified framework for various clinical tasks, including tumor detection and classification. We detail the observation, action, and reward specifications tailored for the histopathological image analysis domain and provide an open-source Python-based interface for both clinicians and researchers. To accommodate different clinical demands, we offer various scenarios for different organs and cancers, including both WSI-based and selected region-based scenarios, showcasing several noteworthy results.
Language Models Show Stable Value Orientations Across Diverse Role-Plays
Lee, Bruce W., Lee, Yeongheon, Cho, Hyunsoo
We demonstrate that large language models (LLMs) exhibit consistent value orientations despite adopting diverse personas, revealing a persistent inertia in their responses that remains stable across the variety of roles they are prompted to assume. To systematically explore this phenomenon, we introduce the role-play-at-scale methodology, which involves prompting LLMs with randomized, diverse personas and analyzing the macroscopic trend of their responses. Unlike previous works that simply feed these questions to LLMs as if testing human subjects, our role-play-at-scale methodology diagnoses inherent tendencies in a systematic and scalable manner by: (1) prompting the model to act in different random personas and (2) asking the same question multiple times for each random persona. This approach reveals consistent patterns in LLM responses across diverse role-play scenarios, indicating deeply encoded inherent tendencies. Our findings contribute to the discourse on value alignment in foundation models and demonstrate the efficacy of role-play-at-scale as a diagnostic tool for uncovering encoded biases in LLMs.