Government
Quantitative Technology Forecasting: a Review of Trend Extrapolation Methods
Tsai, Peng-Hung, Berleant, Daniel, Segall, Richard S., Aboudja, Hyacinthe, Batthula, Venkata Jaipal R., Duggirala, Sheela, Howell, Michael
Quantitative technology forecasting uses quantitative methods to understand and project technological changes. It is a broad field encompassing many different techniques and has been applied to a vast range of technologies. A widely used approach in this field is trend extrapolation. Based on the publications available to us, there has been little or no attempt made to systematically review the empirical evidence on quantitative trend extrapolation techniques. This study attempts to close this gap by conducting a systematic review of technology forecasting literature addressing the application of quantitative trend extrapolation techniques. We identified 25 studies relevant to the objective of this research and classified the techniques used in the studies into different categories, among which growth curves and time series methods were shown to remain popular over the past decade, while newer methods, such as machine learning-based hybrid models, have emerged in recent years. As more effort and evidence are needed to determine if hybrid models are superior to traditional methods, we expect to see a growing trend in the development and application of hybrid models to technology forecasting.
How Do Pedestrians' Perception Change toward Autonomous Vehicles during Unmarked Midblock Multilane Crossings: Role of AV Operation and Signal Indication
Zou, Fengjiao, Ogle, Jennifer Harper, Gerard, Patrick, Jin, Weimin
How Do Pedestrians' Perception Change toward Autonomous Vehicles during Unmarked Midblock Multilane Crossings: Role of AV Operation and Signal Indication ABSTRACT One of the primary impediments hindering the widespread acceptance of autonomous vehicles (AVs) among pedestrians is their limited comprehension of AVs. This study employs virtual reality (VR) to provide pedestrians with an immersive environment for engaging with and comprehending AVs during unmarked midblock multilane crossings. Diverse AV driving behaviors were modeled to exhibit negotiation behavior with a yellow signal indication or nonyielding behavior with a blue signal indication. This paper aims to investigate the impact of various factors, such as AV behavior and signaling, pedestrian past behavior, etc., on pedestrians' perception change of AVs. Before and after the VR experiment, participants completed surveys assessing their perception of AVs, focusing on two main aspects: "Attitude" and "System Effectiveness." The Wilcoxon signed-rank test results demonstrated that both pedestrians' overall attitude score toward AVs and trust in the effectiveness of AV systems significantly increased following the VR experiment. Notably, individuals who exhibited a greater trust in the yellow signals were more inclined to display a higher attitude score toward AVs and to augment their trust in the effectiveness of AV systems. This indicates that the design of the yellow signal instills pedestrians with greater confidence in their interactions with AVs. Further, pedestrians who exhibit more aggressive crossing behavior are less likely to change their perception towards AVs as compared to those pedestrians with more positive crossing behaviors. It is concluded that integrating this paper's devised AV behavior and signaling within an immersive VR setting facilitated pedestrian engagement with AVs, thereby changing their perception of AVs. INTRODUCTION Autonomous vehicles (AVs) can reduce driver involvement starting from level 3 of conditional driving automation, and at level 5 of full driving automation, they can eliminate driver involvement altogether, as per (1).
DEM: A Method for Certifying Deep Neural Network Classifier Outputs in Aerospace
Katz, Guy, Levy, Natan, Refaeli, Idan, Yerushalmi, Raz
Software development in the aerospace domain requires adhering to strict, high-quality standards. While there exist regulatory guidelines for commercial software in this domain (e.g., ARP-4754 and DO-178), these do not apply to software with deep neural network (DNN) components. Consequently, it is unclear how to allow aerospace systems to benefit from the deep learning revolution. Our work here seeks to address this challenge with a novel, output-centric approach for DNN certification. Our method employs statistical verification techniques, and has the key advantage of being able to flag specific inputs for which the DNN's output may be unreliable -- so that they may be later inspected by a human expert. To achieve this, our method conducts a statistical analysis of the DNN's predictions for other, nearby inputs, in order to detect inconsistencies. This is in contrast to existing techniques, which typically attempt to certify the entire DNN, as opposed to individual outputs. Our method uses the DNN as a black-box, and makes no assumptions about its topology. We hope that this work constitutes another step towards integrating DNNs in safety-critical applications -- especially in the aerospace domain, where high standards of quality and reliability are crucial.
DCR-Consistency: Divide-Conquer-Reasoning for Consistency Evaluation and Improvement of Large Language Models
Cui, Wendi, Zhang, Jiaxin, Li, Zhuohang, Damien, Lopez, Das, Kamalika, Malin, Bradley, Kumar, Sricharan
Evaluating the quality and variability of text generated by Large Language Models (LLMs) poses a significant, yet unresolved research challenge. Traditional evaluation methods, such as ROUGE and BERTScore, which measure token similarity, often fail to capture the holistic semantic equivalence. This results in a low correlation with human judgments and intuition, which is especially problematic in high-stakes applications like healthcare and finance where reliability, safety, and robust decision-making are highly critical. This work proposes DCR, an automated framework for evaluating and improving the consistency of LLM-generated texts using a divide-conquer-reasoning approach. Unlike existing LLM-based evaluators that operate at the paragraph level, our method employs a divide-and-conquer evaluator (DCE) that breaks down the paragraph-to-paragraph comparison between two generated responses into individual sentence-to-paragraph comparisons, each evaluated based on predefined criteria. To facilitate this approach, we introduce an automatic metric converter (AMC) that translates the output from DCE into an interpretable numeric score. Beyond the consistency evaluation, we further present a reason-assisted improver (RAI) that leverages the analytical reasons with explanations identified by DCE to generate new responses aimed at reducing these inconsistencies. Through comprehensive and systematic empirical analysis, we show that our approach outperforms state-of-the-art methods by a large margin (e.g., +19.3% and +24.3% on the SummEval dataset) in evaluating the consistency of LLM generation across multiple benchmarks in semantic, factual, and summarization consistency tasks. Our approach also substantially reduces nearly 90% of output inconsistencies, showing promise for effective hallucination mitigation.
ICE-GRT: Instruction Context Enhancement by Generative Reinforcement based Transformers
Zheng, Chen, Sun, Ke, Tang, Da, Ma, Yukun, Zhang, Yuyu, Xi, Chenguang, Zhou, Xun
The emergence of Large Language Models (LLMs) such as ChatGPT and LLaMA encounter limitations in domain-specific tasks, with these models often lacking depth and accuracy in specialized areas, and exhibiting a decrease in general capabilities when fine-tuned, particularly analysis ability in small sized models. To address these gaps, we introduce ICE-GRT, utilizing Reinforcement Learning from Human Feedback (RLHF) grounded in Proximal Policy Optimization (PPO), demonstrating remarkable ability in in-domain scenarios without compromising general task performance. Our exploration of ICE-GRT highlights its understanding and reasoning ability to not only generate robust answers but also to provide detailed analyses of the reasons behind the answer. This capability marks a significant progression beyond the scope of Supervised Fine-Tuning models. The success of ICE-GRT is dependent on several crucial factors, including Appropriate Data, Reward Size Scaling, KL-Control, Advantage Normalization, etc. The ICE-GRT model exhibits state-of-the-art performance in domain-specific tasks and across 12 general Language tasks against equivalent size and even larger size LLMs, highlighting the effectiveness of our approach. We provide a comprehensive analysis of the ICE-GRT, underscoring the significant advancements it brings to the field of LLM.
Uncertainty in GNN Learning Evaluations: A Comparison Between Measures for Quantifying Randomness in GNN Community Detection
Leeney, William, McConville, Ryan
Graph Neural Networks (GNNs) have gained popularity as a neural network-based approach for handling graph-structured data, leveraging their capacity to merge two information sources through the propagation and aggregation of node feature encodings along the network's connectivity [17]. Nodes within a network can be organized into communities based on similarities in associated features and/or edge density [33]. Analyzing the network structure to identify clusters or communities of nodes proves valuable in addressing real-world issues like misinformation detection [25], genomic feature discovery [4], and social network or research recommendation [49]. We consider unsupervised neural approaches to community detection that do not use any ground truth or labels during training to optimise the loss functions. As an unsupervised task, the identification of node clusters relies on latent patterns within the dataset rather than on "ground-truth" labels.
UstanceBR: a multimodal language resource for stance prediction
Pereira, Camila, Pavan, Matheus, Yoon, Sungwon, Ramos, Ricelli, Costa, Pablo, Cavalheiro, Lais, Paraboni, Ivandre
This work introduces UstanceBR, a multimodal corpus in the Brazilian Portuguese Twitter domain for target-based stance prediction. The corpus comprises 86.8 k labelled stances towards selected target topics, and extensive network information about the users who published these stances on social media. In this article we describe the corpus multimodal data, and a number of usage examples in both in-domain and zero-shot stance prediction based on text- and network-related information, which are intended to provide initial baseline results for future studies in the field.
Learning Homogenization for Elliptic Operators
Bhattacharya, Kaushik, Kovachki, Nikola, Rajan, Aakila, Stuart, Andrew M., Trautner, Margaret
Multiscale partial differential equations (PDEs) arise in various applications, and several schemes have been developed to solve them efficiently. Homogenization theory is a powerful methodology that eliminates the small-scale dependence, resulting in simplified equations that are computationally tractable while accurately predicting the macroscopic response. In the field of continuum mechanics, homogenization is crucial for deriving constitutive laws that incorporate microscale physics in order to formulate balance laws for the macroscopic quantities of interest. However, obtaining homogenized constitutive laws is often challenging as they do not in general have an analytic form and can exhibit phenomena not present on the microscale. In response, data-driven learning of the constitutive law has been proposed as appropriate for this task. However, a major challenge in data-driven learning approaches for this problem has remained unexplored: the impact of discontinuities and corner interfaces in the underlying material. These discontinuities in the coefficients affect the smoothness of the solutions of the underlying equations. Given the prevalence of discontinuous materials in continuum mechanics applications, it is important to address the challenge of learning in this context; in particular, to develop underpinning theory that establishes the reliability of data-driven methods in this scientific domain. The paper addresses this unexplored challenge by investigating the learnability of homogenized constitutive laws for elliptic operators in the presence of such complexities. Approximation theory is presented, and numerical experiments are performed which validate the theory in the context of learning the solution operator defined by the cell problem arising in homogenization for elliptic PDEs.
Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspective
Wu, Baoyuan, Zhu, Zihao, Liu, Li, Liu, Qingshan, He, Zhaofeng, Lyu, Siwei
Adversarial machine learning (AML) studies the adversarial phenomenon of machine learning, which may make inconsistent or unexpected predictions with humans. Some paradigms have been recently developed to explore this adversarial phenomenon occurring at different stages of a machine learning system, such as backdoor attack occurring at the pre-training, in-training and inference stage; weight attack occurring at the post-training, deployment and inference stage; adversarial attack occurring at the inference stage. However, although these adversarial paradigms share a common goal, their developments are almost independent, and there is still no big picture of AML. In this work, we aim to provide a unified perspective to the AML community to systematically review the overall progress of this field. We firstly provide a general definition about AML, and then propose a unified mathematical framework to covering existing attack paradigms. According to the proposed unified framework, we build a full taxonomy to systematically categorize and review existing representative methods for each paradigm. Besides, using this unified framework, it is easy to figure out the connections and differences among different attack paradigms, which may inspire future researchers to develop more advanced attack paradigms. Finally, to facilitate the viewing of the built taxonomy and the related literature in adversarial machine learning, we further provide a website, \ie, \url{http://adversarial-ml.com}, where the taxonomies and literature will be continuously updated.
Survey on Publicly Available Sinhala Natural Language Processing Tools and Research
Sinhala is the native language of the Sinhalese people who make up the largest ethnic group of Sri Lanka. The language belongs to the globe-spanning language tree, Indo-European. However, due to poverty in both linguistic and economic capital, Sinhala, in the perspective of Natural Language Processing tools and research, remains a resource-poor language which has neither the economic drive its cousin English has nor the sheer push of the law of numbers a language such as Chinese has. A number of research groups from Sri Lanka have noticed this dearth and the resultant dire need for proper tools and research for Sinhala natural language processing. However, due to various reasons, these attempts seem to lack coordination and awareness of each other. The objective of this paper is to fill that gap of a comprehensive literature survey of the publicly available Sinhala natural language tools and research so that the researchers working in this field can better utilize contributions of their peers. As such, we shall be uploading this paper to arXiv and perpetually update it periodically to reflect the advances made in the field.