Finding "Gems" in Big Data

In 1945, Count Richard Taaffe*, a Dublin gem collector, was sorting through a set of spinel gems that he had bought, and found one that refracted light differently - instead of simply bending light rays, it split them into two rays ("double refraction"). The anomalous gem was named after him and earned a place on the "world's rarest gems" list. In analytics, it sometimes not the rule (i.e. the model) that is of interest, but rather the exception. Detecting anomalous cases in large datasets is critical in conducting surveillance, countering credit-card fraud, protecting against network hacking, combating insurance fraud, and many more applications in government, business and healthcare. The techniques of anomaly detection are not new to the era of Big Data. Dr. Nitin Indurkhya, who teaches the Anomaly Detection course at Statistics.com, told me of an interesting application of anomaly detection to data that long pre-dates the era of Big Data.