The smallest eigenvalues and the associated eigenvectors (i.e., eigenpairs) of a graph Laplacian matrix have been widely used in spectral clustering and community detection. However, in real-life applications the number of clusters or communities (say, $K$) is generally unknown a-priori. Consequently, the majority of the existing methods either choose $K$ heuristically or they repeat the clustering method with different choices of $K$ and accept the best clustering result. The first option, more often, yields suboptimal result, while the second option is computationally expensive. In this work, we propose an incremental method for constructing the eigenspectrum of the graph Laplacian matrix. This method leverages the eigenstructure of graph Laplacian matrix to obtain the $K$-th smallest eigenpair of the Laplacian matrix given a collection of all previously computed $K-1$ smallest eigenpairs. Our proposed method adapts the Laplacian matrix such that the batch eigenvalue decomposition problem transforms into an efficient sequential leading eigenpair computation problem. As a practical application, we consider user-guided spectral clustering. Specifically, we demonstrate that users can utilize the proposed incremental method for effective eigenpair computation and for determining the desired number of clusters based on multiple clustering metrics.

Anomaly detection is a very pervasive problem applicable to a variety of domains including network intrusion, fraud detection, and system failures. It is a crucial task in many applications because failure to detect anomalous activity could result in highly undesirable outcomes. For example, (i) detection of anomalous medical claims is important to identify fraud; (ii) detection of fraudulent credit card transactions is necessary to help prevent identity theft; and (iii) detection of abnormal network traffic is necessary to identify hacking. Many techniques have been developed for anomaly detection. These methods can be broadly classified into two categories: (i) rule-based systems, and (ii) statistical datadriven approaches. The rule-based systems are based on domain expertise and look for specific types of anomalies while the data-driven approaches look to identify anomalies by identifying statistically rare patterns. Examples of datadriven methods include parametric methods that assume a known family for the nominal (non-anomalous) distribution and nonparametric methods such as those using unsupervised or semi-supervised support vector machines (SVMs) [1], [2] or based on minimum volume set estimation [3], [4], [5]. The advantage of data-driven approaches over rule-based methods is that they can identify novel types of anomalies that are unknown to the domain expert.

Probabilistic (or Bayesian) modeling and learning offers interesting possibilities for systematic representation of uncertainty using probability theory. However, probabilistic learning often leads to computationally challenging problems. Some problems of this type that were previously intractable can now be solved on standard personal computers thanks to recent advances in Monte Carlo methods. In particular, for learning of unknown parameters in nonlinear state-space models, methods based on the particle filter (a Monte Carlo method) have proven very useful. A notoriously challenging problem, however, still occurs when the observations in the state-space model are highly informative, i.e. when there is very little or no measurement noise present, relative to the amount of process noise. The particle filter will then struggle in estimating one of the basic components for probabilistic learning, namely the likelihood $p($data$|$parameters$)$. To this end we suggest an algorithm which initially assumes that there is substantial amount of artificial measurement noise present. The variance of this noise is sequentially decreased in an adaptive fashion such that we, in the end, recover the original problem or possibly a very close approximation of it. The main component in our algorithm is a sequential Monte Carlo (SMC) sampler, which gives our proposed method a clear resemblance to the SMC^2 method. Another natural link is also made to the ideas underlying the approximate Bayesian computation (ABC). We illustrate it with numerical examples, and in particular show promising results for a challenging Wiener-Hammerstein benchmark problem.

In this post, we consider different approaches for time series modeling. The forecasting approaches using linear models, ARIMA alpgorithm, XGBoost machine learning algorithm are described. Results of different model combinations are shown. For probabilistic modeling the approaches using copulas and Bayesian inference are considered. Time series analysis, especially forecasting, is an important problem of modern predictive analytics.

We just import numpy and matplotlib. I haven't used pandas but you can certainly do. Read this excellent article by Pankajashree R to get started with Pandas. In the second line we slice the data set and save the first column as an array to X. reshape(-1,1) tells python to convert the array into a matrix with one coloumn. "-1" tells python to figure out the rows by itself.

Note: Please access the full code in this GitHub repo. The multitude of methods jointly referred to as "deep learning" have disrupted the fields of machine learning and data science, rendering decades of engineering know-how almost completely irrelevant--or so common opinion would have it. Of all these, one method that stands out in its overwhelming simplicity, robustness, and usefulness is the transfer of learned representations. Especially for computer vision, this approach has brought about unparalleled ability, accessible to practitioners of all levels, and making previously insurmountable tasks as easy as from keras.applications import *. Put simply, the method dictates that a large data set should be used in order to learn to represent the object of interest (image, time-series, customer, even a network) as a feature vector, in a way that lends itself to downstream data science tasks such as classification or clustering.

Here comes the most important group of articles published on AV. These articles explain who can become an analyst, how to become an analyst and how to sharpen your analytical thinking.

Missing values are representative of the messiness of real world data. There can be a multitude of reasons why they occur -- ranging from human errors during data entry, incorrect sensor readings, to software bugs in the data processing pipeline. The normal reaction is frustration. Missing data are probably the most widespread source of errors in your code, and the reason for most of the exception-handling. If you try to remove them, you might reduce the amount of data you have available dramatically -- probably the worst that can happen in machine learning.

Survival analysis refers to a class of statistical techniques that measure the effect of predictors on the time until an event, rather than the probability of an event occurring. As the name indicates, this technique has roots in the field of medical research for evaluating the effect of drugs or medical procedures on time until death. However, there are many less morbid applications of this technique, such as the following business analytics examples that I've observed during my 20 years as a data scientist: The most commonly used survival analysis techniques are Kaplan-Meier and Cox Regression. The Kaplan-Meier test is already widely used within the pharmaceutical industry for clinical drug trials, comparing the effects of drugs and their placebos on either time to recovery or to death. In an article for The New Yorker, Malcolm Gladwell includes an interesting description of the critical role of Kaplan-Meier tests in the search for effective cancer treatments. This type of test determines if there is a statistically significant difference between the survival time of two or more groups.

By now anyone who reads virtually any trade magazine has been hearing incessantly about how machine learning is going to transform their industry in profound ways. Marketers will be able to read potential customers' minds, farms will produce unprecedented yields, doctors will be able to stem diseases before they begin to form. And of course, we've all heard how machine learning will eventually take our jobs. It may very well be said of machine learning that there never have been so many wild predictions made about something which the majority of the public knows so little. So what exactly is machine learning? And what can we reasonably expect in the next ten years?