Analyses performed using Spark of brain activity in a larval zebrafish: embedding dynamics of whole-brain activity into lower-dimensional trajectories. The growth of data volumes in industry and research poses tremendous opportunities, as well as tremendous computational challenges. As data sizes have outpaced the capabilities of single machines, users have needed new systems to scale out computations to multiple nodes. As a result, there has been an explosion of new cluster programming models targeting diverse computing workloads.1,4,7,10 At first, these models were relatively specialized, with new models developed for new workloads; for example, MapReduce4 supported batch processing, but Google also developed Dremel13 for interactive SQL queries and Pregel11 for iterative graph algorithms. In the open source Apache Hadoop stack, systems like Storm1 and Impala9 are also specialized. Even in the relational database world, the trend has been to move away from "one-size-fits-all" systems.18 Unfortunately, most big data applications need to combine many different processing types. The very nature of "big data" is that it is diverse and messy; a typical pipeline will need MapReduce-like code for data loading, SQL-like queries, and iterative machine learning. Specialized engines can thus create both complexity and inefficiency; users must stitch together disparate systems, and some applications simply cannot be expressed efficiently in any engine. In 2009, our group at the University of California, Berkeley, started the Apache Spark project to design a unified engine for distributed data processing. Spark has a programming model similar to MapReduce but extends it with a data-sharing abstraction called "Resilient Distributed Datasets," or RDDs.25 Using this simple extension, Spark can capture a wide range of processing workloads that previously needed separate engines, including SQL, streaming, machine learning, and graph processing2,26,6 (see Figure 1).

The overall goal is to target treatment specifically to each individual so that clinical outcomes for that individual are optimized. One direction of attack is to use patient data to discover decision rules which specify the treatment to use as a function of a vector of features from the patient. Regression and classification are important statistical tools for estimating such rules based on either observational data or data from a randomized trial, and machine learning can help with this because of its ability to artfully handle high dimensional feature spaces with potentially complex interactions.

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We'll also be posting a weekly calendar of upcoming robotics events for the next two months; here's what we have so far (send us your events!): Let us know if you have suggestions for next week, and enjoy today's videos. The 2016 U.S. Robotics Roadmap was released this week; it's a massive document authored by 150 roboticists that's intended to help frame and guide research and policy decisions with the goal of solving societal problems in the United States. We'll be taking a closer look at it, but here's a 30-minute summary from lead editor Henrik Christensen: The legged robot ANYmal can support disaster relief teams with safer search and rescue operations. With its advanced locomotion capabilities, ANYmal can operate in rough outdoor environments, crawl through pipes, and access buildings over steps and stairs.

Many alarms have sounded on the potential for artificial intelligence (AI) technologies to upend the workforce, especially for easy-to-automate jobs. But managers at all levels will have to adapt to the world of smart machines. The fact is, artificial intelligence will soon be able to do the administrative tasks that consume much of managers' time faster, better, and at a lower cost. How can managers -- from the front lines to the C-suite -- thrive in the age of AI? To find out, we surveyed 1,770 managers from 14 countries and interviewed 37 executives in charge of digital transformation at their organizations.

Scientists have created a "Nightmare Machine" - an artificial intelligence system that can understand what makes certain images frightening, and tranform harmless-looking images into stuff of nightmares. The primary reason for building Nightmare Machine was to explore the common fear inspired by intelligent computers, said researchers including Pinar Yanardag from Massachusetts Institute of Technology (MIT) Media Lab. They wanted to confront the anxiety inspired by AI, and simultaneously test if a computer is capable of understanding and visualising what makes people afraid. The designers used "deep learning" - a system that mimics the neural connections in a human brain - to teach a computer what makes for a frightening visual, according to Manuel Cebrian, a principal research scientist at CSIRO in Australia. "Deep-learning algorithms perform remarkably well in several tasks considered difficult or impossible," Cebrian said.

NEC Corporation has launched a new software program that uses artificial intelligence (AI) in video footage search to quickly identify a person by facial recognition. NeoFace Image Data Mining (Idm) is a new product offering from NEC that can use video footage, for example, data gathered by CCTV cameras, and scan it to accurately identify an individual whose image is captured on camera. It can also be used to search for people who appear at a certain time and place, or who appear with other specified individuals. A complete search for a specific person among one million captured images can be concluded in under 10 seconds. Idm combines existing facial recognition technology with profiling parameters – what NEC refers to as'Profiling Across Spatio-Temporal Data' technology.

Just because Halloween is over doesn't mean the fears have ended. A new project uses autonomous computers to discover your deepest nightmares. The project, dubbed "Nightmare Machine," comes from a partnership between Australia and the U.S. The algorithm would let a computer understand what makes certain videos or images scary. One would hope the algorithm would then remove the scary bits and replace them with something more appealing. However, it uses data to transform any photo into something terrifying.

The idea of artificial intelligence (AI) -- autonomous computers that can learn independently -- makes some people extremely uneasy, regardless of what the computers in question might be doing. Those individuals probably wouldn't find it reassuring to hear that a group of researchers is deliberately training computers to get better at scaring people witless. The project, appropriately enough, is named "Nightmare Machine." Digital innovators in the U.S. and Australia partnered to create an algorithm that would enable a computer to understand what makes certain images frightening, and then use that data to transform any photo, no matter how harmless-looking, into the stuff of nightmares. Images created by Nightmare Machine are unsettling, to say the least.

Determining the optimal size and orientation of small-scale residential based PV arrays will become increasingly complex in the future smart grid environment with the introduction of smart meters and dynamic tariffs. However consumers can leverage the availability of smart meter data to conduct a more detailed exploration of PV investment options for their particular circumstances. In this paper, an optimization method for PV orientation and sizing is proposed whereby maximizing the PV investment value is set as the defining objective. Solar insolation and PV array models are described to form the basis of the PV array optimization strategy. A constrained particle swarm optimization algorithm is selected due to its strong performance in non-linear applications. The optimization algorithm is applied to real-world metered data to quantify the possible investment value of a PV installation under different energy retailers and tariff structures. The arrangement with the highest value is determined to enable prospective small-scale PV investors to select the most cost-effective system.

The artificial intelligence (AI) currently being developed is largely benevolent. It can mimic the way humans think, complete menial and repetitive tasks, and more. But that doesn't prevent people from being afraid of AI, thinking it will take away jobs or eventually turn Terminator into a documentary. Somebody thought AI wasn't scary enough, and did something to change that. Researchers from MIT and Australia's CSIRO have created AI that actively warps pictures into scary nightmare fuel.