Avoiding the Midas Touch: Consequences of Misaligned AI Supplementary Material

This document contains theorem proofs and algorithms for Avoiding the Midas Touch: Consequences of Misaligned AI . Some parts of the main text are repeated for completeness. In this section, we formalize the problem presented in the introduction in the context of objective function design for AI agents. If they could simply express the entirety of their preferences to the robot, there would not be value misalignment. Unfortunately, there are many aspects of the world about which the human cares, and it is intractable to enumerate this complete set to the robot.