The state for this agent consists in two binary features called'left' and'right'. When'left' is 1 then'right' is 0, this means that the agent is not aligned with the leader and that the leader is on the left of the agent. When'right' is 1 then'left' is 0, this means that the agent is not aligned with the leader and that the leader is on the right of the agent. When both'left' and'right' are 0, this means that the agent is aligned with the leader. Using matrix manipulation on the filtered image, we can easily extract the following information to determine if'left' is 1 or 0 and if'right' is 1 or 0 by following these rules: The decision is going to be determined based on the one with maximum probability calculated with the output layer with a softmax activation function. If'left' equal to 1 and action is either going straight or right then reward -5, otherwise 5 If'right' equal to 1 and action is either going straight or left then reward -5, otherwise 5 If'right' and'left' equal to 0 and action is not going straight then reward -5, otherwise 5 From this last component, we can see that a simple heuristic would solve the problem because if we know where the leader is the action is easy to define and it will be the same for the same state representation.