performance metric


Walmart patents surveillance tool to listen to employees conversations

Daily Mail

Walmart has raised the ire of privacy advocates with a new patent for an audio surveillance tool. The freshly filed patent describes the need for'sounds sensors' and'listening to the frontend' technology in its stores that can pick up on conversations between employees and customers. Using these recordings, Walmart would identify employees in the audio and study it to measure their performance at the company. Walmart has raised the ire of privacy advocates with a patent for an audio surveillance tool. 'A need exists for ways to capture the sounds resulting from people in the shopping facility and determine performance of employees based on those sounds,' Walmart explains in the patent, which was filed April 20, 2017 but only made public this week.


Eliciting Binary Performance Metrics

arXiv.org Machine Learning

Given a binary prediction problem, which performance metric should the classifier optimize? We address this question by formalizing the problem of metric elicitation. In particular, we focus on eliciting binary performance metrics from pairwise preferences, where users provide relative feedback for pairs of classifiers. By exploiting key properties of the space of confusion matrices, we obtain provably query efficient algorithms for eliciting linear and linear-fractional metrics. We further show that our method is robust to feedback and finite sample noise.


Binary Classification with Karmic, Threshold-Quasi-Concave Metrics

arXiv.org Machine Learning

Complex performance measures, beyond the popular measure of accuracy, are increasingly being used in the context of binary classification. These complex performance measures are typically not even decomposable, that is, the loss evaluated on a batch of samples cannot typically be expressed as a sum or average of losses evaluated at individual samples, which in turn requires new theoretical and methodological developments beyond standard treatments of supervised learning. In this paper, we advance this understanding of binary classification for complex performance measures by identifying two key properties: a so-called Karmic property, and a more technical threshold-quasi-concavity property, which we show is milder than existing structural assumptions imposed on performance measures. Under these properties, we show that the Bayes optimal classifier is a threshold function of the conditional probability of positive class. We then leverage this result to come up with a computationally practical plug-in classifier, via a novel threshold estimator, and further, provide a novel statistical analysis of classification error with respect to complex performance measures.


Deep Reinforcement Learning in Ice Hockey for Context-Aware Player Evaluation

arXiv.org Artificial Intelligence

A variety of machine learning models have been proposed to assess the performance of players in professional sports. However, they have only a limited ability to model how player performance depends on the game context. This paper proposes a new approach to capturing game context: we apply Deep Reinforcement Learning (DRL) to learn an action-value Q function from 3M play-by-play events in the National Hockey League (NHL). The neural network representation integrates both continuous context signals and game history, using a possession-based LSTM. The learned Q-function is used to value players' actions under different game contexts. To assess a player's overall performance, we introduce a novel Game Impact Metric (GIM) that aggregates the values of the player's actions. Empirical Evaluation shows GIM is consistent throughout a play season, and correlates highly with standard success measures and future salary.


A simple 2D CNN for MNIST digit recognition – Towards Data Science

#artificialintelligence

Convolutional Neural Networks (CNNs) are the current state-of-art architecture for image classification task. Whether it is facial recognition, self driving cars or object detection, CNNs are being used everywhere. In this post, a simple 2-D Convolutional Neural Network (CNN) model is designed using keras with tensorflow backend for the well known MNIST digit recognition task. The data set used here is MNIST dataset as mentioned above. The MNIST database (Modified National Institute of Standards and Technology database) is a large database of handwritten digits (0 to 9).


Yet Another Caret Workshop

#artificialintelligence

You should always set the seed before calling train. Probably not the most amazing \(R 2\) value you have ever seen, but that's alright. Note that calling the model fit displays the most crucial information in a succinct way. Let's move on to a classification algorithm. It's good practice to start with a logistic regression and take it from there.


Evaluating Hospital Case Cost Prediction Models Using Azure Machine Learning Studio

arXiv.org Machine Learning

Ability for accurate hospital case cost modelling and prediction is critical for efficient health care financial management and budgetary planning. A variety of regression machine learning algorithms are known to be effective for health care cost predictions. The purpose of this experiment was to build an Azure Machine Learning Studio tool for rapid assessment of multiple types of regression models. The tool offers environment for comparing 14 types of regression models in a unified experiment: linear regression, Bayesian linear regression, decision forest regression, boosted decision tree regression, neural network regression, Poisson regression, Gaussian processes for regression, gradient boosted machine, nonlinear least squares regression, projection pursuit regression, random forest regression, robust regression, robust regression with mm-type estimators, support vector regression. The tool presents assessment results arranged by model accuracy in a single table using five performance metrics. Evaluation of regression machine learning models for performing hospital case cost prediction demonstrated advantage of robust regression model, boosted decision tree regression and decision forest regression. The operational tool has been published to the web and openly available for experiments and extensions.


How To Implement Machine Learning Algorithm Performance Metrics From Scratch With Python

@machinelearnbot

This article was written by Jason Brownlee. Jason is the editor-in-chief at MachineLearningMastery.com.He has a Masters and PhD in Artificial Intelligence, has published books on Machine Learning and has written operational code that is running in production.


Do machines actually beat doctors? ROC curves and performance metrics

@machinelearnbot

Deep learning research in medicine is a bit like the Wild West at the moment; sometimes you find gold, sometimes a giant steampunk spider-bot causes a ruckus. This has derailed my series on whether AI will be replacing doctors soon, as I have felt the need to focus a bit more on how to assess the quality of medical AI research.