Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve