In 2020, we introduced a deep reinforcement learning method capable of generating superhuman chip layouts, which we then published in Nature and open-sourced on GitHub. AlphaChip has inspired an explosion of work on AI for chip design, and has been deployed in state-of-the-art chips across Alphabet and extended by external chipmakers. Even so, a non-peer-reviewed invited paper at ISPD 2023 questioned its performance claims, despite failing to run our method as described in Nature. For example, it did not pre-train the RL method (removing its ability to learn from prior experience), used substantially fewer compute resources (20x fewer RL experience collectors and half as many GPUs), did not train to convergence (standard practice in machine learning), and evaluated on test cases that are not representative of modern chips. Recently, Igor Markov published a meta-analysis of three papers: our peer-reviewed Nature paper, the non-peer-reviewed ISPD paper, and Markov's own unpublished paper (though he does not disclose that he co-authored it). Although AlphaChip has already achieved widespread adoption and impact, we publish this response to ensure that no one is wrongly discouraged from innovating in this impactful area.

A 2021 paper in Nature by Mirhoseini et al.30 about the use of reinforcement learning (RL) in the physical design of silicon chips raised eyebrows, drew critical media coverage, and stirred up controversy due to poorly documented claims. The paper, authored by Google researchers, withheld critical methodological steps, and most inputs needed to reproduce its results. Our meta-analysis shows how two separate evaluations filled in the gaps and demonstrated that Google RL lags behind human chip designers, a well-known algorithm (simulated annealing), and generally available commercial software, while also being slower. Crosschecked data indicates that the integrity of the Nature paper is substantially undermined, owing to errors in conduct, analysis, and reporting. Before publishing, Google rebuffed internal allegations of fraud which still stand.

Hydrogen is one of the most abundant elements in the universe. On Earth, hydrogen is normally a gas. But when it is under high temperatures and pressures--the conditions that exist within many planets, such as Jupiter--hydrogen goes through a series of phase transitions and takes on the properties of a liquid metal. One of the metallic properties it takes on is becoming an electrical conductor. In a new paper in the Nature journal's "Matters Arising," researchers at the University of Rochester Laboratory for Laser Energetics (LLE), including lead author Valentin Karasiev, an LLE staff scientist; graduate student Josh Hinz; and Suxing Hu, an associate professor of mechanical engineering and a distinguished scientist at the LLE, respond to a 2020 Nature paper that used machine learning techniques to study the liquid-liquid phase transitions of dense hydrogen from an insulating liquid to a liquid metal.

In my previous post A First Look at Reinforcement Learning, I attempted to use Deep Q learning to solve the CartPole problem. In this post, I will be further exploring Deep Q learning but in the…

Artificial intelligence has become the driving force for a vibrant round of industrial transformations in China and around the world. Many countries and enterprises see the AI revolution as an excellent opportunity to promote strong domestic economic and technological development. Last month, a group of artificial intelligence pioneers from 12 Chinese AI institutions published the perspective paper Towards a New Generation of Artificial Intelligence in China in the respected journal Nature Machine Intelligence. This is the first such survey on the full scope of AI in China. The paper looks at the New Generation Artificial Intelligence (NGAI) Development Plan of China (2015– 2030), which was published in 2017 as a blueprint for the rapid construction of a complete Chinese AI ecosystem. Many of the new perspective paper's authors participated in different stages of the NGAI Plan, which outlines national strategies for science and technology as well as education, and identifies a range of related challenges to be overcome.

Starting with the Google DeepMind paper, there has been a lot of new attention around training models to play video games. You, the data scientist/engineer/enthusiast, may not work in reinforcement learning but probably are interested in teaching neural networks to play video games. The lessons below were gleaned from working on my own implementation of the Nature paper. The lessons are aimed at people who work with data but may run into some issues with some of the non-standard approaches used in the reinforcement learning community when compared with typical supervised learning use cases. I will address both technical details of the parameters of the neural networks and the libraries involved.

This article was first published in the July 2015 issue of WIRED magazine. Be the first to read WIRED's articles in print before they're posted online, and get your hands on loads of additional content by subscribing online The future of artificial intelligence begins with a game of Space Invaders. From the start, the enemy aliens are making kills -- three times they destroy the defending laser cannon within seconds. Half an hour in, and the hesitant player starts to feel the game's rhythm, learning when to fire back or hide. Finally, after playing ceaselessly for an entire night, the player is not wasting a single bullet, casually shooting the high-score floating mothership in between demolishing each alien. No one in the world can play a better game at this moment. This player, it should be mentioned, is not human, but an algorithm on a graphics processing unit programmed by a company called DeepMind. Instructed simply to maximise the score and fed only the data stream of 30,000 pixels per frame, the algorithm -- known as a deep Q-network – is then given a new challenge: an unfamiliar Pong-like game called Breakout, in which it needs to hit a ball through a rainbow-coloured brick wall. "After 30 minutes and 100 games, it's pretty terrible, but it's learning that it should move the bat towards the ball," explains DeepMind's cofounder and chief executive, a 38-year-old artificial-intelligence researcher named Demis Hassabis. "Here it is after an hour, quantitatively better but still not brilliant. But two hours in, it's more or less mastered the game, even when the ball's very fast. After four hours, it came up with an optimal strategy -- to dig a tunnel round the side of the wall, and send the ball round the back in a superhuman accurate way. The designers of the system didn't know that strategy."

This code should take 2-4 days to complete. The run_nature.py script uses parameters consistent with the Nature paper. The final policies should be better, but it will take 6-10 days to finish training. Either script will store output files in a folder prefixed with the name of the ROM. Pickled version of the network objects are stored after every epoch.