Spark has a number of configuration options for how it controls execution. One important component of this is the settings around locality. Locality, simply put, refers to where data is relative to where data can be processed. Suppose an executor is free, but data would have to be copied across the network, in order to process it. Spark must decide whether it should execute that network transfer, or if instead it should wait for an executor that has local access to the data to become free.

Machine learning models, especially deep learning ones, can be complex. Chi Zeng walks us through how to debug, monitor, and examine the decisions of a TensorFlow-based model using the TensorBoard suite of visualizations. Chi Zeng works on the TensorBoard suite of visualizations within Google Brain. This video was recorded at QCon.ai 2018: bit.ly/2piRtLl The next QCon is in New York, June 25-29, 2018.

When it comes to diagnosing skin cancer, man may actually be worse than machine. A new study from the University of Heidelberg found that a form of artificial intelligence known as a deep learning convolutional neural network (CNN) was able to more accurately identify malignant and benign skin lesions than a fleet of dermatologists. "For the first time we compared a CNN's diagnostic performance with a large international group of 58 dermatologists, including 30 experts," wrote the researchers. "Most dermatologists were outperformed by the CNN. Irrespective of any physicians' experience, they may benefit from assistance by a CNN's image classification."

There is a lot of buzz about the promise of artificial intelligence (AI) and machine learning--from self-driving cars to predicting heart attacks, AI is spreading like wildfire across industries, triggering a massive investment in talent as businesses rush to adopt and/or deploy the technology. But while there is growing excitement around what new applications AI will enable next, the reality is many issues and limitations still exist. Following are several key challenges that must be overcome before we can realize AI's truly amazing applications. The availability of greater volumes and sources of data is enabling capabilities in AI and machine learning. There is a seemingly endless torrent of data being collected, but it remains incredibly hard to integrate that data.

A Google artificial intelligence (AI) expert believes Facebook can be used for "psychological control" and says researchers who want to work for the social media giant should "show some conscience" and abandon the platform. François Chollet, a software engineer who specializes in deep-learning algorithms, said in a series of tweets published that internet users should now strive towards technology products "that are the anti-Facebook." "The problem with Facebook is not just the loss of your privacy and the fact that it can be used as a totalitarian panopticon," he wrote. "The more worrying issue … is its use of digital information consumption as a psychological control vector." He spoke out in the wake of the Cambridge Analytica scandal in which an app developer allegedly harvested the Facebook account information of 50 million users for use in targeted political campaigns.

Skin cancer detection won't be turned over to machines anytime soon, but artificial intelligence detected skin cancer more accurately than a large group of international dermatologists in controlled testing, Agence France Presse reports. In an academic study and clinical trial published in Annals of Oncology, the study's lead author, Professor Holger A. Haenssle, of the University of Heidelberg Department of Dermatology, wrote, "Most dermatologists were outperformed by the CNN. Regardless of any physician's level of experience, they may benefit from assistance by a CNN's image classification." The study pitted 58 dermatologists from 17 countries against a deep learning convolutional neural network (CNN). Prior to the test, researchers from Germany, France, and the U.S. taught the CNN to differentiate benign skin lesions from dangerous melanomas. In the process, the team showed more than 100,000 images of correctly identified skin cancers to the neural network, which was designed with Google's Inception v4 CNN architecture.

I'm reading "Artificial Intelligence -- The Revolution Hasn't Happened Yet" by Berkeley computer science professor Michael I Jordan. It is a quick, light, read but thought provoking. His point is that once the dust settles after the hype around ubiquitous Artificial Intelligence (AI), we'll realize that what is being called AI is less general purpose intelligence than a collection of powerful tools for augmenting human intelligence (e.g., Siri, what he calls intelligence augmentation or IA) and the intelligent infrastructure (II) that makes these tools possible. There are also some good historical anecdotes in the article. Whether or not you agree with Jordan's point in this piece, I'd still encourage you to check out the reading list he's long suggested to his post docs and grad students, it is a gem.

Susceptibility of deep neural networks to adversarial attacks poses a major theoretical and practical challenge. All efforts to harden classifiers against such attacks have seen limited success. Two distinct categories of samples to which deep networks are vulnerable, "adversarial samples" and "fooling samples", have been tackled separately so far due to the difficulty posed when considered together. In this work, we show how one can address them both under one unified framework. We tie a discriminative model with a generative model, rendering the adversarial objective to entail a conflict. Our model has the form of a variational autoencoder, with a Gaussian mixture prior on the latent vector. Each mixture component of the prior distribution corresponds to one of the classes in the data. This enables us to perform selective classification, leading to the rejection of adversarial samples instead of misclassification. Our method inherently provides a way of learning a selective classifier in a semi-supervised scenario as well, which can resist adversarial attacks. We also show how one can reclassify the rejected adversarial samples.

Recent work has developed methods for learning deep network classifiers that are provably robust to norm-bounded adversarial perturbation; however, these methods are currently only possible for relatively small feedforward networks. In this paper, in an effort to scale these approaches to substantially larger models, we extend previous work in three main directions. First, we present a technique for extending these training procedures to much more general networks, with skip connections (such as ResNets) and general nonlinearities; the approach is fully modular, and can be implemented automatically (analogous to automatic differentiation). Second, in the specific case of $\ell_\infty$ adversarial perturbations and networks with ReLU nonlinearities, we adopt a nonlinear random projection for training, which scales linearly in the number of hidden units (previous approaches scaled quadratically). Third, we show how to further improve robust error through cascade models. On both MNIST and CIFAR data sets, we train classifiers that improve substantially on the state of the art in provable robust adversarial error bounds: from 5.8% to 3.1% on MNIST (with $\ell_\infty$ perturbations of $\epsilon=0.1$), and from 80% to 36.4% on CIFAR (with $\ell_\infty$ perturbations of $\epsilon=2/255$). Code for all experiments in the paper is available at https://github.com/locuslab/convex_adversarial/.

We introduce a method for following high-level navigation instructions by mapping directly from images, instructions and pose estimates to continuous low-level velocity commands for real-time control. The Grounded Semantic Mapping Network (GSMN) is a fully-differentiable neural network architecture that builds an explicit semantic map in the world reference frame by incorporating a pinhole camera projection model within the network. The information stored in the map is learned from experience, while the local-to-world transformation is computed explicitly. We train the model using DAggerFM, a modified variant of DAgger that trades tabular convergence guarantees for improved training speed and memory use. We test GSMN in virtual environments on a realistic quadcopter simulator and show that incorporating an explicit mapping and grounding modules allows GSMN to outperform strong neural baselines and almost reach an expert policy performance. Finally, we analyze the learned map representations and show that using an explicit map leads to an interpretable instruction-following model.