Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning models

Mathematical reasoning requires to deploy most of our finer-grained cognitive abilities, including sophisticated pattern recognition skills, language understanding, symbolic processing, and abstract thinking, making it one of the highest achievements of human intellect. It is therefore not surprising that the scientific community has always regarded mathematical and logical reasoning as crucial steps in building intelligent machines (Newell and Simon, 1956; Bundy, 1983). However, although computers excel at crunching numbers, solving mathematical problems remains a formidable challenge for artificial intelligence (Choi, 2021). On the one hand, grounding structured mathematical knowledge into some form of intrinsic meaning is a longstanding problem in symbolic AI (Searle, 1980; Harnad, 1990). On the other hand, neural networks always lagged in learning math, and such limitation has been traditionally considered an essential feature of their very nature, which is rooted on statistical pattern recognition abilities rather than the use of explicit syntactic rules (Fodor and Pylyshyn, 1988; Marcus, 2018).