Investigating high-performance data engineering

Big data has always been a part of high-performance computing and the science it supports, but new open-source technologies are now being applied to a wider range of scientific and business problems. We've spent time recently testing some of the big data toolkits. Two of the largest drivers of big data applications have been mobile applications and the Internet-of-Things (IoT). Smartphones contain a remarkable array of sensors and are very interesting as mobile devices for sensing and recording a user's environment. The increasing prevalence of low-power sensors monitoring air and water quality, traffic density and so on gives an opportunity to take better socio-economic decisions with better data.