the-risk-of-a-i-controlling-its-feedback

Some AIs make choices or learn based on reinforcements given by a "reward" in a process called reinforcement learning where software decides how to maximize such reward. However, this reinforcement could lead to dangerous results. The pathologist William Thompson originally considered what is now known as the reinforcement learning problem in 1933. Given two untested therapies and a population of patients, he wondered how to cure the most patients. For Thompson, choosing a course of therapy was the action, and a patient cured was the reward. The reinforcement learning problem more broadly concerns how to arrange your behaviors to optimally gain rewards over the long run.