In case of multiple variables say X1 and X2, we can create a third new feature (say X3) which is the product of X1 and X2 i.e. Firstly we read the data using read.csv()

Use neighborhood component analysis (NCA) to choose features for machine learning models. Manipulate and analyze data that is too big to fit in memory. Perform support vector machine (SVM) and Naive Bayes classification, create bags of decision trees, and fit lasso regression on out-of-memory data. Manipulate, compare, and store text data efficiently . Develop clients for MATLAB Production Server in any programming language that supports HTTP.

Regardless of whether the learner is a human or machine, the basic learning process is similar. Machine learning algorithms are divided into categories according to their purpose. There are lots of overlaps in which ML algorithms are applied to a particular problem. As a result, for the same problem, there could be many different ML models possible. So, coming out with the best ML model is an art that requires a lot of patience and trial and error.

In this work, we study the robust subspace tracking (RST) problem and obtain one of the first two provable guarantees for it. The goal of RST is to track sequentially arriving data vectors that lie in a slowly changing low-dimensional subspace, while being robust to corruption by additive sparse outliers. It can also be interpreted as a dynamic (time-varying) extension of robust PCA (RPCA), with the minor difference that RST also requires a short tracking delay. We develop a recursive projected compressive sensing algorithm that we call Nearly Optimal RST via ReProCS (ReProCS-NORST) because its tracking delay is nearly optimal. We prove that NORST solves both the RST and the dynamic RPCA problems under weakened standard RPCA assumptions, two simple extra assumptions (slow subspace change and most outlier magnitudes lower bounded), and a few minor assumptions. Our guarantee shows that NORST enjoys a near optimal tracking delay of $O(r \log n \log(1/\epsilon))$. Its required delay between subspace change times is the same, and its memory complexity is $n$ times this value. Thus both these are also nearly optimal. Here $n$ is the ambient space dimension, $r$ is the subspaces' dimension, and $\epsilon$ is the tracking accuracy. NORST also has the best outlier tolerance compared with all previous RPCA or RST methods, both theoretically and empirically (including for real videos), without requiring any model on how the outlier support is generated. This is possible because of the extra assumptions it uses.

Stochastic gradient descent (SGD) and its variations, e.g., SGD with either classic or Nesterov momentum, RMSProp, Adam, adaptive learning rates, etc., are popular in diverse stochastic optimization problems, e.g., machine learning and online signal processing [1]-[6]. Off-the-shelf methods from convex optimizations, e.g., the quasi-Newton method, conjugate gradient method and truncated Newton method, i.e., the Hessian-free optimization, are attracting more attentions [7]-[10], and find many successful applications in stochastic optimizations as well. At the same time, searchings for new optimization theories and learning rules are always active, and methods like natural gradient descent, relative gradient descent, and equilibrated SGD (ESGD) [11]-[13], provide us with great insight into the properties of the parameter spaces and cost function surfaces in stochastic optimizations. This paper studies the performance of preconditioned SGD (PSGD) [14], a method which explicitly considers the non-convexity and gradient noises in stochastic optimizations. We consider preconditioners in several forms, i.e., dense preconditioner, diagonal preconditioner, Kronecker product preconditioner and more flexible forms. ESGD and batch normalization [18] are shown to be PSGDs with specific forms of preconditioners. We also consider different ways to evaluate the Hessianvector product, an important measurement that helps PSGD to adaptively extract the curvature information of cost surfaces. Our benchmark problems have synthetic and real world data, and involves most commonly used neural network models, e.g., recurrent and convolutional networks. We argue that Kronecker product preconditioners are particularly suitable for training neural networks since affine transformations are their basic building blocks.

Multimodal applications offer a challenge to model selection in machine learning, as the interactions between different data modalities (e.g., between video and audio, or between images and questions) may require a complicated prior in order to accurately capture regularities necessary for downstream tasks. The particular multimodal application that we consider in this work is that of visual question-answering (VQA), i.e., of producing a natural language response to the combined input consisting of an image and a natural language question pertaining specifically to that image. In the case of VQA, the complexity of the task exists in both extracting useful feature representations from the question and the image, as well as the "fusion" of these feature representations by combining them in order to predict the answer to the question. In this paper, we focus on the problem of combining feature representations in the VQA task through models based on a generic "fusion operator" definition. We illustrate the complexity of model selection for the VQA task by designing a class of multimodal fusion operators, each of which combines raw question and visual data streams to predict answers to the given questions based on an image. We evaluate specific instances of high performing fusion operators belonging to the same design class. We evaluate and discuss three multimodal fusion architectural components that emerge as improving performance as part of the investigation into the general class of fusion operators: 1) The use of a gating mechanism, wherein individual features extracted by the fusion operator are turned on or off by a multiplicative interaction.

Population risk---the expectation of the loss over the sampling mechanism---is always of primary interest in machine learning. However, learning algorithms only have access to empirical risk, which is the average loss over training examples. Although the two risks are typically guaranteed to be pointwise close, for applications with nonconvex nonsmooth losses (such as modern deep networks), the effects of sampling can transform a well-behaved population risk into an empirical risk with a landscape that is problematic for optimization. The empirical risk can be nonsmooth, and it may have many additional local minima. This paper considers a general optimization framework which aims to find approximate local minima of a smooth nonconvex function $F$ (population risk) given only access to the function value of another function $f$ (empirical risk), which is pointwise close to $F$ (i.e., $\|F-f\|_{\infty} \le \nu$). We propose a simple algorithm based on stochastic gradient descent (SGD) on a smoothed version of $f$ which is guaranteed to find an $\epsilon$-second-order stationary point if $\nu \le O(\epsilon^{1.5}/d)$, thus escaping all saddle points of $F$ and all the additional local minima introduced by $f$. We also provide an almost matching lower bound showing that our SGD-based approach achieves the optimal trade-off between $\nu$ and $\epsilon$, as well as the optimal dependence on problem dimension $d$, among all algorithms making a polynomial number of queries. As a concrete example, we show that our results can be directly used to give sample complexities for learning a ReLU unit, whose empirical risk is nonsmooth.

Learning linear predictors with the logistic loss---both in stochastic and online settings---is a fundamental task in learning and statistics, with direct connections to classification and boosting. Existing "fast rates" for this setting exhibit exponential dependence on the predictor norm, and Hazan et al. (2014) showed that this is unfortunately unimprovable. Starting with the simple observation that the logistic loss is 1-mixable, we design a new efficient improper learning algorithm for online logistic regression that circumvents the aforementioned lower bound with a regret bound exhibiting a doubly-exponential improvement in dependence on the predictor norm. This provides a positive resolution to a variant of the COLT 2012 open problem of McMahan and Streeter (2012) when improper learning is allowed. This improvement is obtained both in the online setting and, with some extra work, in the batch statistical setting with high probability. We also show that the improved dependency on predictor norm is also near-optimal. Leveraging this improved dependency on the predictor norm yields the following applications: (a) we give algorithms for online bandit multiclass learning with the logistic loss with an $\tilde{O}(\sqrt{n})$ relative mistake bound across essentially all parameter ranges, thus providing a solution to the COLT 2009 open problem of Abernethy and Rakhlin (2009), and (b) we give an adaptive algorithm for online multiclass boosting with optimal sample complexity, thus partially resolving an open problem of Beygelzimer et al. (2015) and Jung et al. (2017). Finally, we give information-theoretic bounds on the optimal rates for improper logistic regression with general function classes, thereby characterizing the extent to which our improvement for linear classes extends to other parameteric and even nonparametric settings.

Consider users -- perhaps from the research, intelligence, or recruiting community -- who seek to explore graphical data -- perhaps knowledge graphs or social networks. If the graph is small, it is reasonable for these users to directly explore the data by examining nodes and traversing edges. For larger graphs, or for graphs with noisy edges, it rapidly becomes necessary to algorithmically aid users. The problems that arise in this setting are essentially those of information retrieval and recommendation for graphical data, and are well studied [Hasan and Zaki2011, Blanco et al.2013]: identifying the most important edges, predicting links that do not exist, and the like. The responsiveness of these retrieval systems is critical [Gray and Boehm-Davis2000], and has driven numerous system designs in both hardware [Hong et al.2011] and software [Low et al.2014].

String kernels are attractive data analysis tools for analyzing string data. Among them, alignment kernels are known for their high prediction accuracies in string classifications when tested in combination with SVMs in various applications. However, alignment kernels have a crucial drawback in that they scale poorly due to their quadratic computation complexity in the number of input strings, which limits large-scale applications in practice. We present the first approximation named ESP+SFM for alignment kernels by leveraging a metric embedding named edit-sensitive parsing (ESP) and space-efficient feature maps (SFM) for random Fourier features (RFF) for large-scale string analyses. Input strings are projected into vectors of RFF by leveraging ESP and SFM. Then, SVMs are trained on the projected vectors, which enables to significantly improve the scalability of alignment kernels while preserving their prediction accuracies. We experimentally test ESP+ SFM on its ability to learn SVMs for large-scale string classifications with various massive string data, and we demonstrate the superior performance of ESP+SFM with respect to prediction accuracy, scalability and computation efficiency.