Luminous by Silvia Park and Ode to the Half-Broken by Suzanne Palmer are both thoughtful and well-written science fiction novels, featuring robots in richly realised worlds. But there the similarities end, says Emily H. Wilson Do we relate better to stories about robots with faces and bodies? Robots and whether they will one day deserve to be treated like people - or destroy humanity, or both - have interested writers for well over a century now. In the real world, the robot threat appears to involve the uses of artificial intelligence in misinformation and more direct forms of warfare such as drone attacks. In the world of literature, however, many writers focus on individual robots.

Neural Information Processing Systems http://nips.cc/

We study gradient-based optimization methods obtained by directly discretizing a second-order ordinary differential equation (ODE) related to the continuous limit of Nesterov's accelerated gradient method.

Neural Information Processing Systems http://nips.cc/

The dynamics of gradient-based training in neural networks often exhibit nontrivial structures; hence, understanding them remains a central challenge in theoretical machine learning. In particular, a concept of feature unlearning, in which a neural network progressively loses previously learned features over long training, has gained attention. In this study, we consider the infinite-width limit of a two-layer neural network updated with a large-batch stochastic gradient, then derive differential equations with different time scales, revealing the mechanism and conditions for feature unlearning to occur. Specifically, we utilize the fast-slow dynamics: while an alignment of first-layer weights develops rapidly, the second-layer weights develop slowly. The direction of a flow on a critical manifold, determined by the slow dynamics, decides whether feature unlearning occurs. We give numerical validation of the result, and derive theoretical grounding and scaling laws of the feature unlearning. Our results yield the following insights: (i) the strength of the primary nonlinear term in data induces the feature unlearning, and (ii) an initial scale of the second-layer weights mitigates the feature unlearning.