The field of AI alignment is concerned with AI systems that pursue unintended goals. One commonly studied mechanism by which an unintended goal might arise is specification gaming, in which the designer-provided specification is flawed in a way that the designers did not foresee. However, an AI system may pursue an undesired goal even when the specification is correct, in the case of goal misgeneralization. Goal misgeneralization is a specific form of robustness failure for learning algorithms in which the learned program competently pursues an undesired goal that leads to good performance in training situations but bad performance in novel test situations. We demonstrate that goal misgeneralization can occur in practical systems by providing several examples in deep learning systems across a variety of domains. Extrapolating forward to more capable systems, we provide hypotheticals that illustrate how goal misgeneralization could lead to catastrophic risk. We suggest several research directions that could reduce the risk of goal misgeneralization for future systems.

Ray the adventurer was always eager to try new ideas and directions. He was not afraid to enter murky areas, and he always left them better illuminated. He introduced terms to the AI community such as default logic, closed-world assumption, and cognitive robotics; he opened avenues of theoretical research with new resolution proof methods and logics for nonmonotonic reasoning, diagnosis, and action; and he was the prime mover in the Cognitive Robotics initiative that has led to a whole new program of research. And he was an adventurer in more than just ideas. He frequently traveled to remote locations to add to his extraordinary collection of rare and exotic lepidoptera.