Communications of the ACM


Deep Learning Takes on Translation

Communications of the ACM

Most translation implementations employ an encoder network and a decoder network that are trained as a pair for each choice of source and target language. The first, called the encoder, processes input text from one language to create an evolving fixed-length vector representation of the evolving input. Even at a practical level, the separation of encoder and decoder may help solve the problem of inadequate training data for many language pairs. Even at a practical level, the separation of encoder and decoder may help solve the problem of inadequate training data for many language pairs.


Technical Perspective: Low-depth Arithmetic Circuits

Communications of the ACM

The computations of polynomials (over a field, which we shall throughout assume is of zero or large enough characteristic) using arithmetic operations of addition and multiplication (and possibly division) are of course as natural as the computation of Boolean functions via logical gates, and capture many natural important tasks including Fourier transforms, linear algebra, matrix computations and more generally symbolic algebraic computations arising in many settings. Arithmetic circuits are the natural computational model for understanding the computational complexity of such tasks just like Boolean circuits are for Boolean functions. The presence of algebraic structure and mathematical tools supplied by centuries of work in algebra were a source of hope that understanding arithmetic circuits will be much faster and easier than their Boolean siblings. And while we generally know more about arithmetic circuits, their power is far from understood, and in particular, the arithmetic analog VP vs. VNP of the Boolean P vs. NP problem as formulated by Valiant8 is wide open.


ImageNet Classification with Deep Convolutional Neural Networks

Communications of the ACM

We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. Four years ago, while we were at the University of Toronto, our deep neural network called SuperVision almost halved the error rate for recognizing objects in natural images and triggered an overdue paradigm shift in computer vision. To improve their performance, we can collect larger datasets, learn more powerful models, and use better techniques for preventing overfitting. Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held.


Technical Perspective: What Led Computer Vision to Deep Learning?

Communications of the ACM

We are in the middle of the third wave of interest in artificial neural networks as the leading paradigm for machine learning. The following paper by Krizhevksy, Sutskever and Hinton (henceforth KSH) is the paper most responsible for this third wave. The current wave has been called "deep learning" because of the emphasis on having multiple layers of neurons between the input and the output of the neural network; the main architectural design features, however, remain the same as in the second wave, the 1980s. Central to that era was the publication of the back-propagation algorithm for training multilayer perceptrons by Rumelhart, Hinton and Williams.7 This algorithm, a consequence of the chain rule of calculus, had been noted before, for example, by Werbos.8


Optimization Search Finds a Heart of Glass

Communications of the ACM

In the initial state, the spins are not aligned with each other, but as the material cools, an annealing process leads to the spins slowly becoming aligned as the spins of individual atoms flip up and down. The photonic machine at Stanford finds "approximate or exact solutions to the Ising problem," says Stanford University researcher Peter McMahon. D-Wave Systems, Google, Microsoft, and a number of universities and computing companies are focusing on designs that use quantum effects to drive the annealing process. The result of the work may use experience derived from annealing-based techniques to develop algorithms for classical computers, rather than new analog or quantum architectures.


Potential and Peril

Communications of the ACM

More recently, lethal autonomous weapon systems (LAWS) powered by artificial intelligence (AI) have begun to surface, raising ethical issues about the use of AI and causing disagreement on whether such weapons should be banned in line with international humanitarian laws under the Geneva Convention. The campaign defines three types of robotic weapons: human-in-the-loop weapons, robots that can select targets and deliver force only with a human command; human-on-the-loop weapons, robots that can select targets and deliver force under the oversight of a human operator who can override the robots' actions; and human-out-of-the-loop weapons, robots that are capable of selecting targets and delivering force without any human input or interaction. Reporting on a February 2016 round-table discussion on autonomous weapons, civilian safety, and regulation versus prohibition among AI and robotics developers, Heather Roff, a research scientist in the Global Security Initiative at Arizona State University with research interests in the ethics of emerging military technologies, international humanitarian law, humanitarian intervention, and the responsibility to protect, distinguishes automatic weapons from autonomous weapons. Roff describes initial autonomous weapons as limited learning weapons that are capable both of learning and of changing their sub-goals while deployed, saying, "Where sophisticated automatic weapons are concerned, governments must think carefully about whether these weapons should be deployed in complex environments.


Big Data

Communications of the ACM

For our fourth and final Panel in Print, we invited 2014 ACM A.M. Turing Award recipient MICHAEL STONEBRAKER, 2013 ACM Prize recipient DAVID BLEI, 2007 ACM Prize recipient DAPHNE KOLLER, and ACM Fellow VIPIN KUMAR to discuss trends in big data. MICHAEL STONEBRAKER: Imagine this simple example: you show up at your doctor's office and have an x-ray done and you want the doctor to run a query that shows who else has x-rays that look like yours, what was their diagnosis and what was the morbidity of the patients. Although not perfect, randomized case control, or AB testing, is about as good a tool as we have been able to develop for addressing some of the confounders. What role can big data and machine learning play in helping scientists understand data (for example, in the Human Genome project) and bring forth some potential real-world opportunities in health and medicine?


Responsible Research and Innovation in the Digital Age

Communications of the ACM

At a time when increasingly potent technologies are being developed with the potential to transform society, researchers in all technological fields, including information and communications technology (ICT), are under growing pressure to consider and reflect on the motivations, purposes, and possible consequences associated with their research. Instances of ICT raising concerns abound. For example, along with attention-grabbing headlines that artificial intelligence (AI) could ultimately pose an existential threat to humankind, there are more prosaic, yet strongly felt, social transformations already associated with AI technologies. For example, AI is an increasingly powerful protagonist in the story of how digital technologies are transforming the nature of work, as more types of work are mediated digitally, including how it is allocated, assessed, and rewarded.


Listening to Professional Voices

Communications of the ACM

In Draft 1, the Task Force's suggested modifications reflected the need for members to better understand how computing technologies and artifacts impact the social infrastructure and how they ought to promote the common good. Principle 1.1 has been modified to make this change (that almost all people are now impacted by computing) explicit by adding to the principle "acknowledging that all people are stakeholders in computing and its artifacts." Following John Rawls' difference principle,c we emphasized computing professionals' responsibility toward the least powerful: "When the interest of multiple groups conflict, the needs of the least advantaged should be given increased attention and priority." Principle 1.4 also speaks against bullying, a form of harassment based on a power differential rather than on sexual difference (although sexual harassment may also include power differentials).


Making Chips Smarter

Communications of the ACM

In recent years, graphical processing units (GPUs) have become the technology of choice for supporting the neural networks that support AI, deep learning, and machine learning. These include improvements in GPUs as well as work on other technologies such as field programmable gate arrays (FPGAs), Tensor Processing Units (TPUs), and other chip systems and architectures that match specific AI and machine learning requirements. These initiatives, says Bryan Catanzaro, vice president of Applied Deep Learning Research at Nvidia, point in the same general direction: "The objective is to build computation platforms that deliver the performance and energy efficiency needed to build AI with a level of accuracy that isn't possible today." The Nvidia Tesla P100 chip, which packs 15 billion transistors into a silicon chip, delivers extremely high throughput on AI workloads associated with deep learning.