Language acquisition is a special kind of learning problem because the outcome of learning of one generation is the input for the next. That makes it possible for languages to adapt to the particularities of the learner. In this paper, I show that this type of language change has important consequences for models of the evolution and acquisition of syntax. For both artificial systems and non-human animals, learning the syntax of natural languages is a notoriously hard problem. All healthy human infants, in contrast, learn any of the approximately 6000 human languages rapidly, accurately and spon(cid:173) taneously.

But just how smart are these large language models? On the last day of the conference, I interviewed legendary linguist Noam Chomsky, now 93 years old, and Gary Marcus, an emeritus professor of cognitive science at New York University who has spent much of the past decade highlighting the limits of deep learning. Both were distinctly unimpressed with today's cutting edge A.I. Chomsky's big disappointment is that these large language models don't tell us anything at all about how the human brain works. Chomsky has devoted much of his life to advancing the theory that there is a universal grammar, or at least a set of structural concepts, that underpin all human languages, and that this grammar is somehow hard-wired into the brain. Chomsky thinks this explains why human infants can master language so easily--whereas today's computer systems need to be fed what Chomsky rightly calls "astronomical amounts of data" and even then still don't actually understand language at all.

Unlike the carefully scripted dialogue found in most books and movies, the language of everyday interaction tends to be messy and incomplete, full of false starts, interruptions, and people talking over each other. From casual conversations between friends to bickering between siblings to formal discussions in a boardroom, authentic conversation is chaotic. It seems miraculous that anyone can learn a language at all, given the haphazard nature of the linguistic experience. For this reason, many language scientists -- including Noam Chomsky, a founder of modern linguistics -- believe that language learners require a kind of glue to rein in the unruly nature of everyday language. And that glue is grammar: a system of rules for generating grammatical sentences.

"You can't go to a physics conference and say: I've got a great theory. It accounts for everything and is so simple it can be captured in two words: "Anything goes."" Every now and then engineers make an advance, and scientists and lay people begin to ponder the question of whether that advance might yield important insight into the human mind. Descartes wondered whether the mind might work on hydraulic principles; throughout the second half of the 20th century, many wondered whether the digital computer would offer a natural metaphor for the mind. The latest hypothesis to attract notice, both within the scientific community, and in the world at large, is the notion that a technology that is popular today, known as large language models, such as OpenAI's GPT-3, might offer important insight into the mechanics of the human mind. Enthusiasm for such models has grown rapidly; OpenAI's Chief Science Officer Ilya Sutskever recently suggested that such systems could conceivably be "slightly conscious".

For the past few weeks, I've been engaged in an email exchange with my favourite anarcho-syndicalist Noam Chomsky. I reached out to him initially to ask whether recent developments in ANNs (artificial neural networks) had caused him to reconsider his famous linguistic theory Universal Grammar. Our conversation touched on the possible limitations of Deep Learning, how well ANNs really model biological brains and also meandered into more philosophical territory. I'm not going to quote Professor Chomsky directly in this article as our discussion was informal but I will attempt to summarise the key take-aways. Noam Chomsky is first and foremost a professor of linguistics (considered by many to be "the father of modern linguistics") but he is probably better known outside of academic circles as an activist, philosopher and historian.

Equation discovery, also known as symbolic regression, is a type of automated modeling that discovers scientific laws, expressed in the form of equations, from observed data and expert knowledge. Deterministic grammars, such as context-free grammars, have been used to limit the search spaces in equation discovery by providing hard constraints that specify which equations to consider and which not. In this paper, we propose the use of probabilistic context-free grammars in the context of equation discovery. Such grammars encode soft constraints on the space of equations, specifying a prior probability distribution on the space of possible equations. We show that probabilistic grammars can be used to elegantly and flexibly formulate the parsimony principle, that favors simpler equations, through probabilities attached to the rules in the grammars. We demonstrate that the use of probabilistic, rather than deterministic grammars, in the context of a Monte-Carlo algorithm for grammar-based equation discovery, leads to more efficient equation discovery. Finally, by specifying prior probability distributions over equation spaces, the foundations are laid for Bayesian approaches to equation discovery.

When a bilingual student learns to solve word problems in math, we expect the student to be able to solve these problem in both languages the student is fluent in,even if the math lessons were only taught in one language. However, current representations in machine learning are language dependent. In this work, we present a method to decouple the language from the problem by learning language agnostic representations and therefore allowing training a model in one language and applying to a different one in a zero shot fashion. We learn these representations by taking inspiration from linguistics and formalizing Universal Grammar as an optimization process (Chomsky, 2014; Montague, 1970). We demonstrate the capabilities of these representations by showing that the models trained on a single language using language agnostic representations achieve very similar accuracies in other languages.

About two weeks ago, I saw an article (actually, one of my colleagues posted it on our intranet) from the MIT Technology Review about the limitations of Artificial Intelligence. The article is here for those of you who want to read it in full, but the fundamental concept is; while AI has made great strides in the last 20 years or so (see the recent win by Google's AlphaGo over Lee Sodol, who is thought to be one of the best Go players of all time), it is still fundamentally inadequate in one respect – we have not yet built a machine that can carry on a conversation with anything remotely approximating human facility. Quite simply, the computer does not understand the meaning of words that it is using and is therefore unable to use them intelligently. The reason for this, according to the article, is that "words often have meaning based on context and the appearance of the letters and words." It's not enough to be able to identify a concept represented by a bunch of letters strung together.

This article was originally published by Scientific American. The idea that we have brains hardwired with a mental template for learning grammar -- famously espoused by Noam Chomsky of the Massachusetts Institute of Technology -- has dominated linguistics for almost half a century. Recently, though, cognitive scientists and linguists have abandoned Chomsky's "universal grammar" theory in droves because of new research examining many different languages -- and the way young children learn to understand and speak the tongues of their communities. That work fails to support Chomsky's assertions. The research suggests a radically different view, in which learning of a child's first language does not rely on an innate grammar module. Instead the new research shows that young children use various types of thinking that may not be specific to language at all -- such as the ability to classify the world into categories (people or objects, for instance) and to understand the relations among things. These capabilities, coupled with a unique hu man ability to grasp what others intend to communicate, allow language to happen. The new findings indicate that if researchers truly want to understand how children, and others, learn languages, they need to look outside of Chomsky's theory for guidance.

The idea that we have brains hardwired with a mental template for learning grammar--famously espoused by Noam Chomsky of the Massachusetts Institute of Technology--has dominated linguistics for almost half a century. Recently, though, cognitive scientists and linguists have abandoned Chomsky's "universal grammar" theory in droves because of new research examining many different languages--and the way young children learn to understand and speak the tongues of their communities. That work fails to support Chomsky's assertions. The research suggests a radically different view, in which learning of a child's first language does not rely on an innate grammar module. Instead the new research shows that young children use various types of thinking that may not be specific to language at all--such as the ability to classify the world into categories (people or objects, for instance) and to understand the relations among things. These capabilities, coupled with a unique hu man ability to grasp what others intend to communicate, allow language to happen. The new findings indicate that if researchers truly want to understand how children, and others, learn languages, they need to look outside of Chomsky's theory for guidance. This conclusion is important because the study of language plays a central role in diverse disciplines--from poetry to artificial intelligence to linguistics itself; misguided methods lead to questionable results.