We consider the problem of training a model under the presence of label noise. Current approaches identify samples with potentially incorrect labels and reduce their influence on the learning process by either assigning lower weights to them or completely removing them from the training set. In the first case the model however still learns from noisy labels; in the latter approach, good training data can be lost. In this paper, we propose an iterative semi-supervised mechanism for robust learning which excludes noisy labels but is still able to learn from the corresponding samples. To this end, we add an unsupervised loss term that also serves as a regularizer against the remaining label noise. We evaluate our approach on common classification tasks with different noise ratios. Our robust models outperform the state-of-the-art methods by a large margin. Especially for very large noise ratios, we achieve up to 20 % absolute improvement compared to the previous best model.

Recovering sparse conditional independence graphs from data is a fundamental problem in machine learning with wide applications. A popular formulation of the problem is an $\ell_1$ regularized maximum likelihood estimation. Many convex optimization algorithms have been designed to solve this formulation to recover the graph structure. Recently, there is a surge of interest to learn algorithms directly based on data, and in this case, learn to map empirical covariance to the sparse precision matrix. However, it is a challenging task in this case, since the symmetric positive definiteness (SPD) and sparsity of the matrix are not easy to enforce in learned algorithms, and a direct mapping from data to precision matrix may contain many parameters. We propose a deep learning architecture, GLAD, which uses an Alternating Minimization (AM) algorithm as our model inductive bias, and learns the model parameters via supervised learning. We show that GLAD learns a very compact and effective model for recovering sparse graph from data.

In this paper, we propose a novel method for generating a synthetic dataset obeying Gaussian distribution. Compared to the commonly used benchmark datasets with unknown distribution, the synthetic dataset has an explicit distribution, i.e., Gaussian distribution. Meanwhile, it has the same characteristics as the benchmark dataset MNIST. As a result, we can easily apply Deep Neural Networks (DNNs) on the synthetic dataset. This synthetic dataset provides a novel experimental tool to verify the proposed theories of deep learning.

Despite being first proposed about 60 years ago [1], only in the past few years have artificial neural networks (ANNs) become the de facto standard machine learning model [2] achieving accurate state-of-the-art results for a wide range of problems ranging from image classification [3]-[5], object detection [6], [7], semantic segmentation [8], [9], face recognition [10], [11], and text recognition [12], [13], to speech recognition [14]-[16], natural language processing problems such as machine translation [17], [18], language modeling [19], and question answering [20]. This has resulted in a huge industry-wide adoption from leading technology companies such as Google, Facebook, Microsoft, IBM, Yahoo!, Twitter, Adobe, and a quickly growing number of startups. One of the prominent reasons for this recent revival is that in order for ANNs to achieve such performance they need very large labeled datasets and huge computational power at a scale that only recently came into the hands of individual researchers in the form of GPUs [21], which kick-started the deep learning revolution in 2012 [3]. Since then, the trend for demanding more computation and more power consumption for such applications has largely increased. Despite being initially bio-inspired architectures, ANNs have significant differences from actual biological neurons in how computations are performed by neurons, their structure (connection patterns and topologies of neurons), learning (how neurons adapt themselves to new observations), and communication (how inter-neuron data is encoded and passed). One of the main differences of ANNs compared to biological neurons, is how communication is done. While biological neurons use asynchronous trains of spikes in an event-based, data-driven manner that adapts locally to its external stimulation pattern to communicate and encode data (though the specific encoding mechanism used by neurons is not totally understood), ANNs communicate in dense, continuous valued activations, which means that all ANN neurons are working at the same time, thus using lots of computation and energy to operate. Spiking neural networks leverage the benefit from biological neurons to communicate asynchronously in trains of spikes. Thus, spiking neural networks incorporate the concept of time, and instead of all neurons firing at the same time as the case with ANNs, in spiking neural networks neurons fire only when thier intrinsic potential (i.e.

Here, weak labels refer to labels that indicate one deep learning, relies on the availability of abundant, or more events are present in the sample, although do not quality data. In this paper we develop a contain the information as to the event frequency nor the novel framework that maximises the utility of exact location of occurrence(s) (illustrated in Section 2.2, time-series datasets that contain only small quantities Figure 1). Our goal therefore is to improve classification of expertly-labelled data, larger quantities performance in domains with variable quality datasets. of weakly (or coarsely) labelled data and a large volume of unlabelled data. This represents scenarios Our key contribution is as follows. We propose a framework commonly encountered in the real world, that combines the strengths of both traditional algorithms such as in crowd-sourcing applications. In our and deep learning methods, to perform multi-resolution work, we use a nested loop using a Kernel Density Bayesian bootstrapping. We obtain probabilistic labels for Estimator (KDE) to super-resolve the abundant pseudo-fine labels, generated from weak labels, which can low-quality data labels, thereby enabling effective then be used to train a neural network. For the label refinement training of a Convolutional Neural Network from weak to fine we use a Kernel Density Estimator (CNN). We demonstrate two key results: (KDE).

Deep neural networks (DNNs) have recently achieved state-of-the-art performance and provide significant progress in many machine learning tasks, such as image classification, speech processing, natural language processing, etc. However, recent studies have shown that DNNs are vulnerable to adversarial attacks. For instance, in the image classification domain, adding small imperceptible perturbations to the input image is sufficient to fool the DNN and to cause misclassification. The perturbed image, called \textit{adversarial example}, should be visually as close as possible to the original image. However, all the works proposed in the literature for generating adversarial examples have used the $L_{p}$ norms ($L_{0}$, $L_{2}$ and $L_{\infty}$) as distance metrics to quantify the similarity between the original image and the adversarial example. Nonetheless, the $L_{p}$ norms do not correlate with human judgment, making them not suitable to reliably assess the perceptual similarity/fidelity of adversarial examples. In this paper, we present a database for visual fidelity assessment of adversarial examples. We describe the creation of the database and evaluate the performance of fifteen state-of-the-art full-reference (FR) image fidelity assessment metrics that could substitute $L_{p}$ norms. The database as well as subjective scores are publicly available to help designing new metrics for adversarial examples and to facilitate future research works.

This article explores the concepts of ocean wave multivariate multistep forecasting, reconstruction and feature selection. We introduce recurrent neural network frameworks, integrated with Bayesian hyperparameter optimization and Elastic Net methods. We consider both short- and long-term forecasts and reconstruction, for significant wave height and output power of the ocean waves. Sequence-to-sequence neural networks are being developed for the first time to reconstruct the missing characteristics of ocean waves based on information from nearby wave sensors. Our results indicate that the Adam and AMSGrad optimization algorithms are the most robust ones to optimize the sequence-to-sequence network. For the case of significant wave height reconstruction, we compare the proposed methods with alternatives on a well-studied dataset. We show the superiority of the proposed methods considering several error metrics. We design a new case study based on measurement stations along the east coast of the United States and investigate the feature selection concept. Comparisons substantiate the benefit of utilizing Elastic Net. Moreover, case study results indicate that when the number of features is considerable, having deeper structures improves the performance.

In label-noise learning, \textit{noise transition matrix}, denoting the probabilities that clean labels flip into noisy labels, plays a central role in building \textit{statistically consistent classifiers}. Existing theories have shown that the transition matrix can be learned by exploiting \textit{anchor points} (i.e., data points that belong to a specific class almost surely). However, when there are no anchor points, the transition matrix will be poorly learned, and those current consistent classifiers will significantly degenerate. In this paper, without employing anchor points, we propose a \textit{transition-revision} ($T$-Revision) method to effectively learn transition matrices, leading to better classifiers. Specifically, to learn a transition matrix, we first initialize it by exploiting data points that are similar to anchor points, having high \textit{noisy class posterior probabilities}. Then, we modify the initialized matrix by adding a \textit{slack variable}, which can be learned and validated together with the classifier by using noisy data. Empirical results on benchmark-simulated and real-world label-noise datasets demonstrate that without using exact anchor points, the proposed method is superior to the state-of-the-art label-noise learning methods.

The learning of deep models, in which a numerous of parameters are superimposed, is known to be a fairly sensitive process and should be carefully done through a combination of several techniques that can help to stabilize it. We introduce an additional challenge that has never been explicitly studied: the heterogeneity of sparsity at the instance level due to missing values or the innate nature of the input distribution. We confirm experimentally on the widely used benchmark datasets that this variable sparsity problem makes the output statistics of neurons unstable and makes the learning process more difficult by saturating non-linearities. We also provide the analysis of this phenomenon, and based on our analysis, we present a simple technique to prevent this issue, referred to as Sparsity Normalization (SN). Finally, we show that the performance can be significantly improved with SN on certain popular benchmark datasets, or that similar performance can be achieved with lower capacity. Especially focusing on the collaborative filtering problem where the variable sparsity issue has been completely ignored, we achieve new state-of-the-art results on Movielens 100k and 1M datasets, by simply applying Sparsity Normalization (SN).

Attention networks show promise for both vision and language tasks, by emphasizing relationships between constituent elements through appropriate weighting functions. Such elements could be regions in an image output by a region proposal network, or words in a sentence, represented by word embedding. Thus far, however, the learning of attention weights has been driven solely by the minimization of task specific loss functions. We here introduce a method of learning attention weights to better emphasize informative pair-wise relations between entities. The key idea is to use a novel center-mass cross entropy loss, which can be applied in conjunction with the task specific ones. We then introduce a focused attention backbone to learn these attention weights for general tasks. We demonstrate that the focused attention module leads to a new state-of-the-art for the recovery of relations in a relationship proposal task. Our experiments show that it also boosts performance for diverse vision and language tasks, including object detection, scene categorization and document classification.