World Bank hosted its poverty prediction competition on the competition hosting website drivendata.org. The link to the competition is here. We decided to try out our Machine Learning skills on this dataset. Most regular work in ParallelDots is around three themes: Visual Analytics on images and videos, Healthcare AI and NLP, all three of which are solved using Deep Learning techniques. This competition was a chance to try out something new and build our internal codebase to handle tabular datasets like what we had in the competition.

Modern artificial intelligence is way beyond playing chess; it has mastered Go and kicks butt in Dota 2, among other games. What started as a test-lab monkey has evolved into something akin to a prodigy child. Artificial intelligence, or AI, may still have to be fed information, but once it has gathered enough, it can come up with results that mimic the original data. First came the static images -- AI managed to create perfectly convincing images of people who have never existed. Then it showed it was perfectly capable of mimicking different seasons.

What Tomorrow's Business Leaders Need to Know About Machine Learning Sometimes I write a blog just to formulate and organize a point of view, and I think it's time that I pull together the bounty of excellent information about Machine Learning. This is a topic with which business leaders must become comfortable, especially tomorrow's business leaders (tip for my next semester University of San Francisco business students!). Machine learning is a key capability that will help organizations drive optimization and monetization opportunities, and there have been some recent developments that will place basic machine learning capabilities into the hands of the lines of business. By the way, there is an absolute wealth of freely-available material on machine learning, so I've included a sources section at the end of this blog for folks who want more details on machine learning. Time to dive into the world of machine learning!

The randomized experiment is an important tool for inferring the causal impact of an intervention. The recent literature on statistical learning methods for heterogeneous treatment effects demonstrates the utility of estimating the marginal conditional average treatment effect (MCATE), i.e., the average treatment effect for a subpopulation of respondents who share a particular subset of covariates. However, each proposed method makes its own set of restrictive assumptions about the intervention's effects, the underlying data generating processes, and which subpopulations (MCATEs) to explicitly estimate. Moreover, the majority of the literature provides no mechanism to identify which subpopulations are the most affected--beyond manual inspection--and provides little guarantee on the correctness of the identified subpopulations. Therefore, we propose Treatment Effect Subset Scan (TESS), a new method for discovering which subpopulation in a randomized experiment is most significantly affected by a treatment. We frame this challenge as a pattern detection problem where we maximize a nonparametric scan statistic (measurement of distributional divergence) over subpopulations, while being parsimonious in which specific subpopulations to evaluate. Furthermore, we identify the subpopulation which experiences the largest distributional change as a result of the intervention, while making minimal assumptions about the intervention's effects or the underlying data generating process. In addition to the algorithm, we demonstrate that the asymptotic Type I and II error can be controlled, and provide sufficient conditions for detection consistency---i.e., exact identification of the affected subpopulation. Finally, we validate the efficacy of the method by discovering heterogeneous treatment effects in simulations and in real-world data from a well-known program evaluation study.

We need machine learning for tasks that are too complex for humans to code directly, i.e. tasks that are so complex that it is impractical, if not impossible, for us to work out all of the nuances and code for them explicitly. So instead, we provide a machine learning algorithm with a large amount of data and let it explore and search for a model that will work out what the programmers have set out to achieve. Let's look at these two examples: Then comes the Machine Learning Approach: instead of writing a program by hand for each specific task, we collect lots of examples that specify the correct output for a given input. A machine learning algorithm then takes these examples and produces a program that does the job. The program produced by the learning algorithm may look very different from a typical hand-written program -- it may contain millions of numbers. If we do it right, the program works for new cases, as well as the ones we trained it on. If the data changes, the program can change too by training from the new data. You should note that conducting massive amounts of computation is now cheaper than paying someone to write a task-specific program.

In 1900, 30 million people in the United States were farmers. By 1990 that number had fallen to under 3 million even as the population more than tripled. So, in a matter of speaking, 90% of American agriculture workers lost their jobs, mostly due to automation. Yet somehow, the 20th century was still seen as an era of unprecedented prosperity. In the decades to come, we are likely to see similar shifts.

There are tons of resources to help you learn the different aspects of R, and as a beginner this can be overwhelming. It's also a dynamic language and rapidly changing, so it's important to keep up with the latest tools and technologies. That's why R-bloggers and DataCamp have worked together to bring you a learning path for R. Each section points you to relevant resources and tools to get you started and keep you engaged to continue learning. Just like R, this learning path is a dynamic resource.

A fatal crash that occurred when an autonomous SUV operated by Uber struck and killed a pedestrian could have better been avoided if a human was in control of the vehicle, some experts believe. Footage of the incident, which occurred on Sunday in Tempe, Arizona, and resulted in the death of 49-year-old Elaine Herzberg, was released by the local police Wednesday. Experts have suggested that Uber's self-driving technology should have been able to avoid the crash and failed to do so. Experts believe a human driver could have avoided a fatal accident involving Uber's self-driving SUV. The video includes footage from a dashboard camera showing a view outside the car, as well as a view of the operator employed by Uber sitting behind the wheel of the vehicle and take over if the autonomous system does not work as intended.

Machine Learning (ML) is one of the most exciting and dynamic areas of modern research and application. The purpose of this review is to provide an introduction to the core concepts and tools of machine learning in a manner easily understood and intuitive to physicists. The review begins by covering fundamental concepts in ML and modern statistics such as the bias-variance tradeoff, overfitting, regularization, and generalization before moving on to more advanced topics in both supervised and unsupervised learning. Topics covered in the review include ensemble models, deep learning and neural networks, clustering and data visualization, energy-based models (including MaxEnt models and Restricted Boltzmann Machines), and variational methods. Throughout, we emphasize the many natural connections between ML and statistical physics. A notable aspect of the review is the use of Python notebooks to introduce modern ML/statistical packages to readers using physics-inspired datasets (the Ising Model and Monte-Carlo simulations of supersymmetric decays of proton-proton collisions). We conclude with an extended outlook discussing possible uses of machine learning for furthering our understanding of the physical world as well as open problems in ML where physicists maybe able to contribute. (Notebooks are available at https://physics.bu.edu/~pankajm/MLnotebooks.html )

In domains with high knowledge distribution a natural objective is to create principle foundations for collaborative interactive learning environments. We present a first mathematical characterization of a collaborative learning group, a consortium, based on closure systems of attribute sets and the well-known attribute exploration algorithm from formal concept analysis. To this end, we introduce (weak) local experts for subdomains of a given knowledge domain. These entities are able to refute and potentially accept a given (implicational) query for some closure system that is a restriction of the whole domain. On this we build up a consortial expert and show first insights about the ability of such an expert to answer queries. Furthermore, we depict techniques on how to cope with falsely accepted implications and on combining counterexamples. Using notions from combinatorial design theory we further expand those insights as far as providing first results on the decidability problem if a given consortium is able to explore some target domain. Applications in conceptual knowledge acquisition as well as in collaborative interactive ontology learning are at hand.