Goto

Collaborating Authors

 vacuity






Review for NeurIPS paper: Multifaceted Uncertainty Estimation for Label-Efficient Deep Learning

Neural Information Processing Systems

Weaknesses: I have several concerns regarding this work. Firstly, I'm not entirely convinced by the need to introduce an evidence-based Dempster Schafer / Subjective logic framework. The proposed decomposition into vacuity and dissonance is essentially the same thing as decomposing into epistemic uncertainty and aleatoric uncertainty. Why not consider the tractable closed form mutual information decomposition into total, epistemic and aleatoric uncertainties which was derived in previous work? I believe that decomposition would, broadly speaking, have many of the same quantities.


Uncertainty-Aware Reliable Text Classification

Hu, Yibo, Khan, Latifur

arXiv.org Artificial Intelligence

Deep neural networks have significantly contributed to the success in predictive accuracy for classification tasks. However, they tend to make over-confident predictions in real-world settings, where domain shifting and out-of-distribution (OOD) examples exist. Most research on uncertainty estimation focuses on computer vision because it provides visual validation on uncertainty quality. However, few have been presented in the natural language process domain. Unlike Bayesian methods that indirectly infer uncertainty through weight uncertainties, current evidential uncertainty-based methods explicitly model the uncertainty of class probabilities through subjective opinions. They further consider inherent uncertainty in data with different root causes, vacuity (i.e., uncertainty due to a lack of evidence) and dissonance (i.e., uncertainty due to conflicting evidence). In our paper, we firstly apply evidential uncertainty in OOD detection for text classification tasks. We propose an inexpensive framework that adopts both auxiliary outliers and pseudo off-manifold samples to train the model with prior knowledge of a certain class, which has high vacuity for OOD samples. Extensive empirical experiments demonstrate that our model based on evidential uncertainty outperforms other counterparts for detecting OOD examples. Our approach can be easily deployed to traditional recurrent neural networks and fine-tuned pre-trained transformers.


Multidimensional Uncertainty-Aware Evidential Neural Networks

Hu, Yibo, Ou, Yuzhe, Zhao, Xujiang, Cho, Jin-Hee, Chen, Feng

arXiv.org Artificial Intelligence

Traditional deep neural networks (NNs) have significantly contributed to the state-of-the-art performance in the task of classification under various application domains. However, NNs have not considered inherent uncertainty in data associated with the class probabilities where misclassification under uncertainty may easily introduce high risk in decision making in real-world contexts (e.g., misclassification of objects in roads leads to serious accidents). Unlike Bayesian NN that indirectly infer uncertainty through weight uncertainties, evidential NNs (ENNs) have been recently proposed to explicitly model the uncertainty of class probabilities and use them for classification tasks. An ENN offers the formulation of the predictions of NNs as subjective opinions and learns the function by collecting an amount of evidence that can form the subjective opinions by a deterministic NN from data. However, the ENN is trained as a black box without explicitly considering inherent uncertainty in data with their different root causes, such as vacuity (i.e., uncertainty due to a lack of evidence) or dissonance (i.e., uncertainty due to conflicting evidence). By considering the multidimensional uncertainty, we proposed a novel uncertainty-aware evidential NN called WGAN-ENN (WENN) for solving an out-of-distribution (OOD) detection problem. We took a hybrid approach that combines Wasserstein Generative Adversarial Network (WGAN) with ENNs to jointly train a model with prior knowledge of a certain class, which has high vacuity for OOD samples. Via extensive empirical experiments based on both synthetic and real-world datasets, we demonstrated that the estimation of uncertainty by WENN can significantly help distinguish OOD samples from boundary samples. WENN outperformed in OOD detection when compared with other competitive counterparts.


Credibility-limited Base Revision: New Classes and Their Characterizations

Garapa, Marco (Universidade da Madeira) | Fermé, Eduardo | Reis, Maurício

Journal of Artificial Intelligence Research

In this paper we study a kind of operator --known as credibility-limited base revisions-- which addresses two of the main issues that have been pointed out to the AGM model of belief change. Indeed, on the one hand, these operators are defined on belief bases (rather than belief sets) and, on the other hand, they are constructed with the underlying idea that not all new information is accepted. We propose twenty different classes of credibilitylimited base revision operators and obtain axiomatic characterizations for each of them. Additionally we thoroughly investigate the interrelations (in the sense of inclusion) among all those classes. More precisely, we analyse whether each one of those classes is or is not (strictly) contained in each of the remaining ones.


Quantifying Classification Uncertainty using Regularized Evidential Neural Networks

Zhao, Xujiang, Ou, Yuzhe, Kaplan, Lance, Chen, Feng, Cho, Jin-Hee

arXiv.org Artificial Intelligence

Traditional deep neural nets (NNs) have shown the state-of-the-art performance in the task of classification in various applications. However, NNs have not considered any types of uncertainty associated with the class probabilities to minimize risk due to misclassification under uncertainty in real life. Unlike Bayesian neural nets indirectly infering uncertainty through weight uncertainties, evidential neural networks (ENNs) have been recently proposed to support explicit modeling of the uncertainty of class probabilities. It treats predictions of an NN as subjective opinions and learns the function by collecting the evidence leading to these opinions by a deterministic NN from data. However, an ENN is trained as a black box without explicitly considering different types of inherent data uncertainty, such as vacuity (uncertainty due to a lack of evidence) or dissonance (uncertainty due to conflicting evidence). This paper presents a new approach, called a {\em regularized ENN}, that learns an ENN based on regularizations related to different characteristics of inherent data uncertainty. Via the experiments with both synthetic and real-world datasets, we demonstrate that the proposed regularized ENN can better learn of an ENN modeling different types of uncertainty in the class probabilities for classification tasks.