Bayesian Optimisation (BO) is a technique used in optimising a $D$-dimensional function which is typically expensive to evaluate. While there have been many successes for BO in low dimensions, scaling it to high dimensions has been notoriously difficult. Existing literature on the topic are under very restrictive settings. In this paper, we identify two key challenges in this endeavour. We tackle these challenges by assuming an additive structure for the function. This setting is substantially more expressive and contains a richer class of functions than previous work. We prove that, for additive functions the regret has only linear dependence on $D$ even though the function depends on all $D$ dimensions. We also demonstrate several other statistical and computational benefits in our framework. Via synthetic examples, a scientific simulation and a face detection problem we demonstrate that our method outperforms naive BO on additive functions and on several examples where the function is not additive.

Max-norm regularizer has been extensively studied in the last decade as it promotes an effective low-rank estimation for the underlying data. However, such max-norm regularized problems are typically formulated and solved in a batch manner, which prevents it from processing big data due to possible memory budget. In this paper, hence, we propose an online algorithm that is scalable to large-scale setting. Particularly, we consider the matrix decomposition problem as an example, although a simple variant of the algorithm and analysis can be adapted to other important problems such as matrix completion. The crucial technique in our implementation is to reformulating the max-norm to an equivalent matrix factorization form, where the factors consist of a (possibly overcomplete) basis component and a coefficients one. In this way, we may maintain the basis component in the memory and optimize over it and the coefficients for each sample alternatively. Since the memory footprint of the basis component is independent of the sample size, our algorithm is appealing when manipulating a large collection of samples. We prove that the sequence of the solutions (i.e., the basis component) produced by our algorithm converges to a stationary point of the expected loss function asymptotically. Numerical study demonstrates encouraging results for the efficacy and robustness of our algorithm compared to the widely used nuclear norm solvers.

Creating an accurate simulation of the mind is no easy task, and while it took brilliant minds decades to advance us to where we're at right now, we are still ways off our final goal. It is therefore imperative to have more research carried out in this multidisciplinary field, taking in help from researchers in biology, neuroscience, computer science, but also mathematics, physics, chemistry and imaging, in order to speed up this process and tip the scales in our favor for the upcoming decades. This annual review hopes to provide the required information for anyone who is considering this domain as his future endeavor. The reviews will be tackling relatively global characteristics at first in order to familiarize the reader with the basic foundations, and will be getting progressively more specific and in tune with current research in the upcoming parts.

Welcome back to our series of articles sponsored by Intel – "Ask a Data Scientist." Once a week you'll see reader submitted questions of varying levels of technical detail answered by a practicing data scientist – sometimes by me and other times by an Intel data scientist. Think of this new insideBIGDATA feature as a valuable resource for you to get up to speed in this flourishing area of technology. If you have a big data question you'd like answered, please just enter a comment below, or send an e-mail to me at: daniel@insidehpc.com. This week's question is from a reader who wants an explanation of the "bias vs. variance tradeoff in statistical learning."

Hyperparameter optimization is a common problem in machine learning. Machine learning algorithms, from logistic regression to neural nets, depend on well tuned hyperparameters to reach maximum effectiveness. Different hyperparameter optimization strategies have varied performance and cost (in time, money, and compute cycles.) So how do you choose? Evaluating optimization strategies is non-intuitive.

Hi there, my name is Harrison and I frequently do Python programming tutorials on PythonProgramming.net and YouTube.com/sentdex. I do my best to produce tutorials for beginner-intermediate programmers, mainly by making sure nothing is left to abstraction and hand waving. The most recent series is an in-depth machine learning course, aimed at breaking down the complex ML concepts that are typically just "done for you" in a hand-wavy fashion with packages and modules. The machine learning series is aimed at just about anyone with a basic understanding of Python programming and the willingness to learn. If you're confused about something we're doing, I can either help, or point you towards a tutorial that I've done already (I have about 1,000) to help.

Machine learning is the process of building analytical models to automatically discover previously unknown patterns from data that indicate associations, sequences, anomalies (outliers), classifications, and clusters and segments. These patterns reveal hidden rules as to why an event happened--for example, rules that predict likely customer churn. The widely used Cross Industry Standard Process for Data Mining (CRISP-DM) methodology is used to develop predictive analytical models. CRISP-DM includes six phases: business understanding, data understanding, data preparation, model development using supervised and unsupervised learning, model evaluation and model deployment. The business understanding phase involves defining the business problem or use case, the business objectives and the business questions that need to be answered.

Logistic Regression is one of the most powerful classification methods within machine learning and can be used for a wide variety of tasks. Think of pre-policing or predictive analytics in health; it can be used to aid tuberculosis patients, aid breast cancer diagnosis, etc. Think of modeling urban growth, analysing mortgage pre-payments and defaults, forecasting the direction and strength of stock market movement, and even predicting sport outcomes. Reading all of this, the theory[1] of Maximum Entropy Classification might look difficult. In my experience, the average Developer does not believe they can design a proper Maximum Entropy / Logistic Regression Classifier from scratch. I strongly disagree: not only is the mathematics behind is relatively simple, it can also be implemented with a few lines of code.

Most existing learning to hash methods assume that there are sufficient data, either labeled or unlabeled, on the domain of interest (i.e., the target domain) for training. However, this assumption cannot be satisfied in some real-world applications. To address this data sparsity issue in hashing, inspired by transfer learning, we propose a new framework named Transfer Hashing with Privileged Information (THPI). Specifically, we extend the standard learning to hash method, Iterative Quantization (ITQ), in a transfer learning manner, namely ITQ+. In ITQ+, a new slack function is learned from auxiliary data to approximate the quantization error in ITQ. We developed an alternating optimization approach to solve the resultant optimization problem for ITQ+. We further extend ITQ+ to LapITQ+ by utilizing the geometry structure among the auxiliary data for learning more precise binary codes in the target domain. Extensive experiments on several benchmark datasets verify the effectiveness of our proposed approaches through comparisons with several state-of-the-art baselines.

When the training data in a two-class classification problem is overwhelmed by one class, most classification techniques fail to correctly identify the data points belonging to the underrepresented class. We propose Similarity-based Imbalanced Classification (SBIC) that learns patterns in the training data based on an empirical similarity function. To take the imbalanced structure of the training data into account, SBIC utilizes the concept of absent data, i.e. data from the minority class which can help better find the boundary between the two classes. SBIC simultaneously optimizes the weights of the empirical similarity function and finds the locations of absent data points. As such, SBIC uses an embedded mechanism for synthetic data generation which does not modify the training dataset, but alters the algorithm to suit imbalanced datasets. Therefore, SBIC uses the ideas of both major schools of thoughts in imbalanced classification: Like cost-sensitive approaches SBIC operates on an algorithm level to handle imbalanced structures; and similar to synthetic data generation approaches, it utilizes the properties of unobserved data points from the minority class. The application of SBIC to imbalanced datasets suggests it is comparable to, and in some cases outperforms, other commonly used classification techniques for imbalanced datasets.