Chandrasekaran, Parrilo and Willsky (2010) proposed a convex optimization problem to characterize graphical model selection in the presence of unobserved variables. This convex optimization problem aims to estimate an inverse covariance matrix that can be decomposed into a sparse matrix minus a low-rank matrix from sample data. Solving this convex optimization problem is very challenging, especially for large problems. In this paper, we propose two alternating direction methods for solving this problem. The first method is to apply the classical alternating direction method of multipliers to solve the problem as a consensus problem. The second method is a proximal gradient based alternating direction method of multipliers. Our methods exploit and take advantage of the special structure of the problem and thus can solve large problems very efficiently. Global convergence result is established for the proposed methods. Numerical results on both synthetic data and gene expression data show that our methods usually solve problems with one million variables in one to two minutes, and are usually five to thirty five times faster than a state-of-the-art Newton-CG proximal point algorithm.

Legislative behavior centers around the votes made by lawmakers. These votes are captured in roll call data, a matrix with lawmakers in the rows and proposed legislation in the columns. We illustrate a sample of roll call votes for the United States Senate in Figure 1. The seminal work of Poole and Rosenthal (1985) introduced the ideal point model, using roll call data to infer the latent political positions of the lawmakers. The ideal point model is a latent factor model of binary data and an application of item-response theory (Lord 1980) to roll call data. It gives each lawmaker a latent political position along a single dimension and then uses these points (called the ideal points) in a model of the votes.

In binary-transaction data-mining, traditional frequent itemset mining often produces results which are not straightforward to interpret. To overcome this problem, probability models are often used to produce more compact and conclusive results, albeit with some loss of accuracy. Bayesian statistics have been widely used in the development of probability models in machine learning in recent years and these methods have many advantages, including their abilities to avoid overfitting. In this paper, we develop two Bayesian mixture models with the Dirichlet distribution prior and the Dirichlet process (DP) prior to improve the previous non-Bayesian mixture model developed for transaction dataset mining. We implement the inference of both mixture models using two methods: a collapsed Gibbs sampling scheme and a variational approximation algorithm. Experiments in several benchmark problems have shown that both mixture models achieve better performance than a non-Bayesian mixture model. The variational algorithm is the faster of the two approaches while the Gibbs sampling method achieves a more accurate results. The Dirichlet process mixture model can automatically grow to a proper complexity for a better approximation. Once the model is built, it can be very fast to query and run analysis on (typically 10 times faster than Eclat, as we will show in the experiment section). However, these approaches also show that mixture models underestimate the probabilities of frequent itemsets. Consequently, these models have a higher sensitivity but a lower specificity.

Given the joint distribution of a set of random variables (in the form of a Markov random field), we consider the problem of selecting a small subset of these variables to observe so as to accurately predict the remaining unobserved variables. We focus here on Gaussian processes(Rasmussen and Williams, 2006) on graphs, i.e., Gaussian Markov random fields(Gaussian MRFs). Our aim in this paper is to give a subset selection algorithm which, given a budget for the number of variables that can be observed, minimizes the expected squared prediction error averaged over all the variables. We are particularly interested in algorithms with provable guarantees on the prediction error. Our main focus is on Gaussian MRFs on trees and other treelike graphs, or to be precise, bounded tree-width graphs--such graphs have been widely studied in the context of inference, see, e.g., Sudderth (2002). We also consider a special class of Gaussian MRFs, called Gaussian free fields (or GFFs), which arise, among others, in computer vision, see, e.g., Szeliski (1990). We first explain the notation we use and formally state our problem before describing how our work relates to previous research.

In areas like Natural Language Processing, data often have multi-view and high dimension. Recently, CCA [8] has been applied to the multi-view setting as a unsupervised dimension reduction method in [7][10][3] with performance guarantee if the data is generated under certain structure. In [7], they assume the high dimensional multi-view data is generated independently conditioning on a low dimensional hidden state (the model structure will be illustrated later in detail). Under this assumption, the low dimensional features provided by CCA won't lose any useful information compared with the original high dimensional features when applied to linear regression. Also, [6] has applied this CCA method to generate a low dimensional vector representation of words which works well in a lot of NLP tasks. The reason for CCA to work well is that the low dimensional hidden state (throughout the paper we'll use k to denote the dimension of hidden state) 1 contains most information for the supervised tasks and by doing CCA, we are able to generate k dimensional estimate of the hidden state from each view as mentioned by [4], or more precisely, we can find all k directions in the high dimensional space of each view that have nonzero correlation with the hidden state via CCA. Only two views are enough to implement the CCA algorithms above (see [7] for detailed introduction about CCA). Despite it's power in dimension reduction, CCA with two views is still not optimal in the sense that it ends up with a hidden state estimator from each view but it's impossible to tell which view is better by only looking at the two views.

Efficient Natural Evolution Strategies (eNES) is a novel alternative to conventional evolutionary algorithms, using the natural gradient to adapt the mutation distribution. Unlike previous methods based on natural gradients, eNES uses a fast algorithm to calculate the inverse of the exact Fisher information matrix, thus increasing both robustness and performance of its evolution gradient estimation, even in higher dimensions. Additional novel aspects of eNES include optimal fitness baselines and importance mixing (a procedure for updating the population with very few fitness evaluations). The algorithm yields competitive results on both unimodal and multimodal benchmarks.

Collective classification models attempt to improve classification performance by taking into account the class labels of related instances. However, they tend not to learn patterns of interactions between classes and/or make the assumption that instances of the same class link to each other (assortativity assumption). Blockmodels provide a solution to these issues, being capable of modelling assortative and disassortative interactions, and learning the pattern of interactions in the form of a summary network. The Supervised Blockmodel provides good classification performance using link structure alone, whilst simultaneously providing an interpretable summary of network interactions to allow a better understanding of the data. This work explores three variants of supervised blockmodels of varying complexity and tests them on four structurally different real world networks.

This paper suggests a learning-theoretic perspective on how synaptic plasticity benefits global brain functioning. We introduce a model, the selectron, that (i) arises as the fast time constant limit of leaky integrate-and-fire neurons equipped with spiking timing dependent plasticity (STDP) and (ii) is amenable to theoretical analysis. We show that the selectron encodes reward estimates into spikes and that an error bound on spikes is controlled by a spiking margin and the sum of synaptic weights. Moreover, the efficacy of spikes (their usefulness to other reward maximizing selectrons) also depends on total synaptic strength. Finally, based on our analysis, we propose a regularized version of STDP, and show the regularization improves the robustness of neuronal learning when faced with multiple stimuli.

Although the standard formulations of prediction problems involve fully-observed and noiseless data drawn in an i.i.d. manner, many applications involve noisy and/or missing data, possibly involving dependence, as well. We study these issues in the context of high-dimensional sparse linear regression, and propose novel estimators for the cases of noisy, missing and/or dependent data. Many standard approaches to noisy or missing data, such as those using the EM algorithm, lead to optimization problems that are inherently nonconvex, and it is difficult to establish theoretical guarantees on practical algorithms. While our approach also involves optimizing nonconvex programs, we are able to both analyze the statistical error associated with any global optimum, and more surprisingly, to prove that a simple algorithm based on projected gradient descent will converge in polynomial time to a small neighborhood of the set of all global minimizers. On the statistical side, we provide nonasymptotic bounds that hold with high probability for the cases of noisy, missing and/or dependent data. On the computational side, we prove that under the same types of conditions required for statistical consistency, the projected gradient descent algorithm is guaranteed to converge at a geometric rate to a near-global minimizer. We illustrate these theoretical predictions with simulations, showing close agreement with the predicted scalings.

Synaptic plasticity seems to be a capital aspect of the dynamics of neural networks. It is about the physiological modifications of the synapse, which have like consequence a variation of the value of the synaptic weight. The information encoding is based on the precise timing of single spike events that is based on the relative timing of the pre- and post-synaptic spikes, local synapse competitions within a single neuron and global competition via lateral connections. In order to classify temporal sequences, we present in this paper how to use a local hebbian learning, spike-timing dependent plasticity for unsupervised competitive learning, preserving self-organizing maps of spiking neurons. In fact we present three variants of self-organizing maps (SOM) with spike-timing dependent Hebbian learning rule, the Leaky Integrators Neurons (LIN), the Spiking_SOM and the recurrent Spiking_SOM (RSSOM) models. The case study of the proposed SOM variants is phoneme classification and word recognition in continuous speech and speaker independent.