On shallow planning under partial observability