Global Bigdata Conference

The more data you have, the more likely you are to hallucinate patterns. Supposedly, the more phone records the NSA looks at, the more likely it is to flag an innocent as a potential terrorist because he accidentally matched a terrorist detection rule. Mining more attributes of the same entities can indeed increase the risk of hallucination, but machine learning experts are very good at keeping it to a minimum. On the other hand, mining more entities with the same set of attributes decreases the risk, because the rules learned from them will have stronger support. And some learning algorithms can find patterns involving multiple entities, which makes them even more robust: a person videotaping the New York City Hall may not be suspicious, and another buying large quantities of ammonium nitrate may not be either; but if the two are in close phone contact, perhaps the FBI should take a look, just to make sure it's not a bomb plot.