Low impact agency: review and discussion

The problem of artificial intelligence safety can be seen as can be seen as ensuring an agent with the power of causing harm chooses to not do so. In the limit, the agent can be powerful enough that causing existential catastrophe is within its limit, and it has incentives to doing so [6], so our task is to guarantee that it chooses not to. A possible approach is penalize changes in the world caused by agent, leading to the agent not causing catastrophe because that leads to large changes in the world[24]. The hope is that this is a relatively easy objective to align the agent with, as opposed to aligning it with the full range of human values. So, our desideratum is that the AI achieves something while doing as little in the world as possible .