We live in an age where logical reasoning matters the most, whether we use apps on smartphones for weather forecast, or software for improving driving. Ever since the invention of computers, we have grown to expect larger-than-life outputs from them. Computers must handle our basic day-to-day chores as well as solve complex problems. The race to make computers smarter has never ceased. But what more we can expect from computers? No matter what, we have partially succeeded in making computers smarter.

There are more than a trillion sensors in the world today and according to some estimates there will be about 50 trillion cameras worldwide within the next 5 years, all collecting data either sporadically or around the clock. With such explosive growth of available data and computing resources, recent advances in machine learning and data analytics have yielded transformative results across diverse scientific disciplines, including image recognition, natural language processing, cognitive science, and genomics. However, in many engineering applications, quality and error-free data is not easy to obtain, e.g., for system dynamics characterized by bifurcations and instabilities, hysteresis, delayed responses, and often irreversible responses. Admittedly, as in all everyday applications, in engineering problems, the volume of data has increased substantially compared to even a decade ago but analyzing big data is expensive and time-consuming. Data-driven methods, which have been enabled in the past decade by the availability of sensors, data storage, and computational resources, are taking center stage across many disciplines (physical and information) of science.

Now, it's clear that if we use a linear activation function (identity activation function), then the Neural Network will output linear output of the input. This loses much of the representational power of the neural network as often times the output that we are trying to predict has a non-linear relationship with the inputs. It can be shown that if we use a linear activation function for a hidden layer and sigmoid function for an output layer, our model becomes logistic regression model. Due to the fact that a composition of two linear functions is linear function, our area of implementing such Neural Network reduces rapidly. Rare implementation example can be solving regression problem in machine learning (where we use linear activation function in hidden layer).

Given a data set D containing millions of data points and a data consumer who is willing to pay for X to train a machine learning (ML) model over D, how should we distribute this X to each data point to reflect its "value"? In this paper, we define the "relative value of data" via the Shapley value, as it uniquely possesses properties with appealing real-world interpretations, such as fairness, rationality and decentralizability. For general, bounded utility functions, the Shapley value is known to be challenging to compute: to get Shapley values for all N data points, it requires O(2 N) model evaluations for exact computation and O(N N) for (ϵ, δ)-approximation. In this paper, we focus on one popular family of ML models relying on K-nearest neighbors (KNN). The most surprising result is that for unweighted KNN classifiers and regressors, the Shapley value of all N data points can be computed, exactly, in O(N N) time -- an exponential improvement on computational complexity!

Given a data set D containing millions of data points and a data consumer who is willing to pay for X to train a machine learning (ML) model over D, how should we distribute this X to each data point to reflect its "value"? In this paper, we define the "relative value of data" via the Shapley value, as it uniquely possesses properties with appealing real-world interpretations, such as fairness, rationality and decentralizability. For general, bounded utility functions, the Shapley value is known to be challenging to compute: to get Shapley values for all N data points, it requires O(2 N) model evaluations for exact computation and O(N N) for (ϵ, δ)-approximation. In this paper, we focus on one popular family of ML models relying on K-nearest neighbors (KNN). The most surprising result is that for unweighted KNN classifiers and regressors, the Shapley value of all N data points can be computed, exactly, in O(N N) time -- an exponential improvement on computational complexity!

In this post, we will take a tour of the most popular machine learning algorithms. It is useful to tour the main algorithms in the field to get a feeling of what methods are available. There are so many algorithms that it can feel overwhelming when algorithm names are thrown around and you are expected to just know what they are and where they fit. I want to give you two ways to think about and categorize the algorithms you may come across in the field. Both approaches are useful, but we will focus in on the grouping of algorithms by similarity and go on a tour of a variety of different algorithm types.

Statistical learning theory is the broad framework for studying the concept of inference in both supervised and unsupervisedmachine learning. Inference covers the entire spectrum of machine learning, from gaining knowledge, making predictions or decisions and constructing models from a set of labeled or unlabeled data. The entire process is stated in a statistical framework, with every assumption stated mathematically as a null or alternative hypothesis. The practical goals of this approach are to make machine learning more precise (reliably reproduceable) and to create new or improved modeling algorithms. This is primarily accomplished by providing a formal, statistical definition of abstract concepts, like learning, generalization, overfitting and performance, then testing these hypotheses one parameter at a time. But in statistical machine learning, the entire process needs to be automated for a computer program to learn from it.

Combining the capabilities of the programmability of networks by SDN and discovering patterns by machine learning are utilized in security, traffic classification, QoS prediction, and network performance and has attracted the attention of researchers. In this work, we propose HyMER: a novel hybrid machine learning framework for traffic aware energy efficient routing in SDN which has supervised and reinforcement learning components. The supervised learning component consists of feature extraction, training, and testing. The reinforcement learning component learns from existing data or from scratch by iteratively interacting with the network environment. The framework is developed on POX controller and is evaluated on Mininet using Abiline, GEANT, and Nobel-Germany real-world topologies and dynamic traffic traces. Experimental results show that the supervised component achieves up to 70% feature size reduction and more than 80% accuracy in parameter prediction. The refine heuristics algorithm increases the accuracy of the prediction to 100% with 14X to 25X speedup as compared to the brute force method. The reinforcement learning module converges from 100 to 275 iterations and converges twice faster if applied on top of the supervised component. Moreover, HyMER achieves up to 10 watts per switch power saving, 30% link saving, 2 hops decrease in average path length.

We prove that the norm version of the adaptive stochastic gradient method (AdaGrad-Norm) achieves a linear convergence rate for a subset of either strongly convex functions or non-convex functions that satisfy the Polyak-Lojasiewicz (PL) inequality. The paper introduces the notion of Restricted Uniform Inequality of Gradients (RUIG), which describes the uniform lower bound for the norm of the stochastic gradients with respect to the distance to the optimal solution. RUIG plays the key role in proving the robustness of AdaGrad-Norm to its hyper-parameter tuning. On top of RUIG, we develop a novel two-stage framework to prove linear convergence of AdaGrad-Norm without knowing the parameters of the objective functions: Stage I: the step-size decrease fast such that it reaches to Stage II; Stage II: the step-size decreases slowly and converges. This framework can likely be extended to other adaptive stepsize algorithms. The numerical experiments show desirable agreement with our theories.

Similarity plays a fundamental role in many areas, including data mining, machine learning, statistics and various applied domains. Inspired by the success of ensemble methods and the flexibility of trees, we propose to learn a similarity kernel called rpf-kernel through random projection forests (rpForests). Our theoretical analysis reveals a highly desirable property of rpf-kernel: far-away (dissimilar) points have a low similarity value while nearby (similar) points would have a high similarity}, and the similarities have a native interpretation as the probability of points remaining in the same leaf nodes during the growth of rpForests. The learned rpf-kernel leads to an effective clustering algorithm--rpfCluster. On a wide variety of real and benchmark datasets, rpfCluster compares favorably to K-means clustering, spectral clustering and a state-of-the-art clustering ensemble algorithm--Cluster Forests. Our approach is simple to implement and readily adapt to the geometry of the underlying data. Given its desirable theoretical property and competitive empirical performance when applied to clustering, we expect rpf-kernel to be applicable to many problems of an unsupervised nature or as a regularizer in some supervised or weakly supervised settings.