Adequate evaluation of an information retrieval system to estimate future performance is a crucial task. Area under the ROC curve (AUC) is widely used to evaluate the generalization of a retrieval system. However, the objective function optimized in many retrieval systems is the error rate and not the AUC value. This paper provides an efficient and effective non-linear approach to optimize AUC using additive regression trees, with a special emphasis on the use of multi-class AUC (MAUC) because multiple relevance levels are widely used in many ranking applications. Compared to a conventional linear approach, the performance of the non-linear approach is comparable on binary-relevance benchmark datasets and is better on multi-relevance benchmark datasets.

This post is made up of a collection of 10 Github repositories consisting in part, or in whole, of IPython (Jupyter) Notebooks, focused on transferring data science and machine learning concepts. This warmup notebook is from postdoctoral researcher Randal Olson, who uses the common Python ecosystem data analysis/machine learning/data science stack to work with the Iris dataset. Aaron Masino has shared a series of very detailed, very technical machine learning IPython Notebook learning resources. From UC Boulder's Research Computing group, this older collection of notebooks (it's from way back in Fall 2013) covers a wide range of material, with an apparent focus on Linux command line-powered data management.

This is a great project undertaken by Jordi Warmenhoven to implement the concepts from the book An Introduction to Statistical Learning with Applications in R by James, Witten, Hastie, Tibshirani (2013) in Python (the book has practical exercises in R, as you may have guessed). The book is freely available in as a PDF, which makes this repo even more attractive to those looking to learn.

If we tackle a supervised learning problem, my advice is to start with the simplest hypothesis space first. I.e., try a linear model such as logistic regression. If this doesn't work "well" (i.e., it doesn't meet our expectation or performance criterion that we defined earlier), I would move on to the next experiment. I would say that random forests are probably THE "worry-free" approach - if such a thing exists in ML: There are no real hyperparameters to tune (maybe except for the number of trees; typically, the more trees we have the better). On the contrary, there are a lot of knobs to be turned in SVMs: Choosing the "right" kernel, regularization penalties, the slack variable, ... Both random forests and SVMs are non-parametric models (i.e., the complexity grows as the number of training samples increases).

I want to develop a machine learning model in R that I can deploy in Java. I want to describe what I've tried, and how it failed, and what my next iteration is so every one here can help guide me. We have a system where a human will examine raw text and assign it meta-data where each piece of meta-data is a separate category. For example, say we had String 1 and available categories A-Z and a human assigned String 1 categories A, B, C, and F. There is a large amount of human QA that happens afterwards to ensure that String 1 either received all the categories it should have and didn't receive any categories it shouldn't have, for example String 1 should not have received the f category. I am tasked with developing a way to automatically detect if a String needs QA after meta-data has been assigned to it.

In today's blog post, we shall look into time series analysis using R package – forecast. Objective of the post will be explaining the different methods available in forecast package which can be applied while dealing with time series analysis/forecasting. A time series is a collection of observations of well-defined data items obtained through repeated measurements over time. For example, measuring the value of retail sales each month of the year would comprise a time series. My data set contains data of Sales of CARS from Jan-2008 to Dec 2013.

Let's say you have to cluster 10 million points, for instance keywords. So, in short, you can perform k-NN (k-nearest neighbors) clustering or some other types of clustering, which typically is O(n 2) or worse, from a computational complexity point of view. Has anyone ever used a clustering method based on sampling? The idea is to start by sampling 1% (or less) of the 100,000,000 entries, and perform clustering on these pairs of keywords, to create a "seed" or "baseline" cluster structure. The next step is to browse sequentially your 10,000,000 keywords, and for each keyword, find the closest cluster from the baseline cluster structure.

This is a good question, and one that seems to get raised time and time again. Myself and a colleague (Sven Crone from Lancaster University in the UK) published a paper on this issue last year in the International Journal of Forecasting. A summary of our findings can also be found in the book "Credit Scoring, Response Modeling and Insurance Rating. There are also some very good papers by G. Weiss from 2004/5 which are highly cited and referenced in our paper/book. What we found was that for some methods of model construction sample imbalance was not an issue at all – not even a tiny amount.

Python & data analytics go hand in hand. Here is a list of 9 Python data analytics libraries. This list is going to be continuously updated here. Pandas is a library written for the Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series.

Word Sense Disambiguation (WSD) aims to determine the meaning of a word in context, and successful approaches are known to benefit many applications in Natural Language Processing. Although supervised learning has been shown to provide superior WSD performance, current sense-annotated corpora do not contain a sufficient number of instances per word type to train supervised systems for all words. While unsupervised techniques have been proposed to overcome this data sparsity problem, such techniques have not outperformed supervised methods. In this paper, we propose a new approach to building semi-supervised WSD systems that combines a small amount of sense-annotated data with information from Word Sense Induction, a fully-unsupervised technique that automatically learns the different senses of a word based on how it is used. In three experiments, we show how sense induction models may be effectively combined to ultimately produce high-performance semi-supervised WSD systems that exceed the performance of state-of-the-art supervised WSD techniques trained on the same sense-annotated data. We anticipate that our results and released software will also benefit evaluation practices for sense induction systems and those working in low-resource languages by demonstrating how to quickly produce accurate WSD systems with minimal annotation effort.