Why squared error?

Someone recently asked on the statistics Stack Exchange why the squared error is used in statistics. This is something I'd been wondering about myself recently, so I decided to take a crack at answering it. The post below is adapted from that answer. It's true that one could choose to use, say, the absolute error instead of the squared error. In fact, the absolute error is often closer to what you "care about" when making predictions from your model.