In the first part of this series, I introduced the Outbrain Click Prediction machine learning competition. That post described some preliminary and important data science tasks like exploratory data analysis and feature engineering performed for the competition, using a Spark cluster deployed on Google Dataproc. In this post, I describe the competition evaluation, the design of my cross-validation strategy and my baseline models using statistics and trees ensembles. In that competition, Kagglers were required to rank recommended ads by decreasing predicted likelihood of being clicked. Sponsored search advertising, contextual advertising, display advertising and real-time bidding auctions have all relied heavily on the ability of learned models to predict ad click–through rates (CTRs) accurately, quickly and reliably.

A fresh scrape from Glassdoor gives us a good idea about what applicants are asked during a data scientist interview at some of the top companies. Unfortunately for us, almost every company has their interviewees sign NDAs. Since Glassdoor allows anonymity, a few brave souls have given us some fantastic examples of what they were asked during the interview process at top companies like Facebook, Google, and Microsoft. If you find yourself unable to answer some of the questions below, consider checking out a course or a book on the subject. If you'd like to share your answer(s) to any of the questions, leave a comment and I'll add the top ones to the post.

Principal components analysis (PCA) is a statistical technique that allows to identify underlying linear patterns in a data set so it can be expressed in terms of other data set of significatively lower dimension without much loss of information. The final data set should be able to explain most of the variance of the original data set by making a variable reduction. The final variables will be named as principal components. The following image depicts the activity diagram that shows each step of the principal components analysis that will be explained in detail later. In order to illustrate the process described in the previous diagram, we are going to make use of the following data set which has two dimensions.

I've split this post into four sections: Machine Learning, NLP, Python, and Math. For future posts, I may create a similar list of books, online videos, and code repos as I'm compiling a growing collection of those resources too. What's the Difference Between Artificial Intelligence, Machine Learning, and Deep Learning?

Which machine learning algorithm should I use?

Neural networks are considered complicated and they are always explained using neurons and a brain function. But we do not need to learn how to brain works to understand Neural networks structure and how they operate. We can look as something people encounter in everyday life more often, like a corporation hierarchy. Let us start with logistic regression. The logistic regression yields values form 0 to 1, and we can consider the process as making a evaluation.

D. Fox Harrell (fox@csail.mit.edu) is Professor of Digital Media in both the Comparative Media Studies Program and the Computer Science and Artificial Intelligence Laboratory at the Massachusetts Institute of Technology, Cambridge MA, and the founder and director of the Imagination, Computation, and Expression Laboratory. Chong-U Lim (culim@csail.mit.edu) recently completed his Ph.D. in electrical engineering and computer science from the Computer Science and Artificial Intelligence Laboratory at the Massachusetts Institute of Technology, Cambridge MA, where he was a member of the Imagination, Computation, and Expression Laboratory.

The paper deals with the adaptation of a new measure for the unsupervised feature selection problems. The proposed measure is based on space filling concept and is called the coverage measure. This measure was used for judging the quality of an experimental space filling design. In the present work, the coverage measure is adapted for selecting the smallest informative subset of variables by reducing redundancy in data. This paper proposes a simple analogy to apply this measure. It is implemented in a filter algorithm for unsupervised feature selection problems. The proposed filter algorithm is robust with high dimensional data and can be implemented without extra parameters. Further, it is tested with simulated data and real world case studies including environmental data and hyperspectral image. Finally, the results are evaluated by using random forest algorithm.

Traditional linear methods for forecasting multivariate time series are not able to satisfactorily model the non-linear dependencies that may exist in non-Gaussian series. We build on the theory of learning vector-valued functions in the reproducing kernel Hilbert space and develop a method for learning prediction functions that accommodate such non-linearities. The method not only learns the predictive function but also the matrix-valued kernel underlying the function search space directly from the data. Our approach is based on learning multiple matrix-valued kernels, each of those composed of a set of input kernels and a set of output kernels learned in the cone of positive semi-definite matrices. In addition to superior predictive performance in the presence of strong non-linearities, our method also recovers the hidden dynamic relationships between the series and thus is a new alternative to existing graphical Granger techniques.

Abstract--Power supply from renewable resources is on a global rise where it is forecasted that renewable generation will surpass other types of generation in a foreseeable future. Increased generation from renewable resources, mainly solar and wind, exposes the power grid to more vulnerabilities, conceivably due to their variable generation, thus highlighting the importance of accurate forecasting methods. This paper proposes a two-stage day-ahead solar forecasting method that breaks down the forecasting into linear and nonlinear parts, determines subsequent forecasts, and accordingly, improves accuracy of the obtained results. To further reduce the error resulted from nonstationarity of the historical solar radiation data, a data processing approach, including pre-process and post-process levels, is integrated with the proposed method. Numerical simulations on three test days with different weather conditions exhibit the effectiveness of the proposed two-stage model. Figure 1 The new added U.S. electric generation from 2010 to Q1 2016 [2].