Ptolemy and the Limits of Deep Learning