How Bayesian Networks are pioneering the 'smart data' revolution

The era of'big data' offers enormous opportunities for societal improvements. There is an expectation – and even excitement – that, by simply applying sophisticated machine learning algorithms to'big data' sets, we may automatically find solutions to problems that were previously either unsolvable or would incur prohibitive economic costs. Yet, the clever algorithms needed to process big data cannot (and will never) solve most of the critical risk analysis problems that we face. Big data, even when carefully collected is typically unstructured and noisy; even the'biggest data' typically lack crucial, often hidden, information about key causal or explanatory variables that generate or influence the data we observe. For example, the world's leading economists failed to predict the 2008–2010 international financial crisis because they relied on models based on historical statistical data that could not adapt to new circumstances, even when those circumstances were foreseeable by contrarian experts.