Transformers achieve unrivalled performance in modelling language, but remain inefficient in terms of memory and time complexity. A possible remedy is to reduce the sequence length in the intermediate layers by pooling fixed-length segments of tokens. Nevertheless, natural units of meaning, such as words or phrases, display varying sizes. To address this mismatch, we equip language models with a dynamic-pooling mechanism, which predicts segment boundaries in an autoregressive fashion. We compare several methods to infer boundaries, including end-to-end learning through stochastic re-parameterisation, supervised learning (based on segmentations from subword tokenizers or spikes in conditional entropy), as well as linguistically motivated boundaries. We perform character-level evaluation on texts from multiple datasets and morphologically diverse languages. The results demonstrate that dynamic pooling, which jointly segments and models language, is both faster and more accurate than vanilla Transformers and fixed-length pooling within the same computational budget.

Ideally, dialogue systems should generate responses that are faithful to the knowledge contained in relevant documents. However, many models generate hallucinated responses instead that contradict it or contain unverifiable information. To mitigate such undesirable behaviour, it has been proposed to fine-tune a `negative expert' on negative examples and subtract its parameters from those of a pre-trained model. However, intuitively, this does not take into account that some parameters are more responsible than others in causing hallucinations. Thus, we propose to weigh their individual importance via (an approximation of) the Fisher Information matrix, which measures the uncertainty of their estimate. We call this method Elastic Weight Removal (EWR). We evaluate our method -- using different variants of Flan-T5 as a backbone language model -- on multiple datasets for information-seeking dialogue generation and compare our method with state-of-the-art techniques for faithfulness, such as CTRL, Quark, DExperts, and Noisy Channel reranking. Extensive automatic and human evaluation shows that EWR systematically increases faithfulness at minor costs in terms of other metrics. However, we notice that only discouraging hallucinations may increase extractiveness, i.e. shallow copy-pasting of document spans, which can be undesirable. Hence, as a second main contribution, we show that our method can be extended to simultaneously discourage hallucinations and extractive responses. We publicly release the code for reproducing EWR and all baselines.

In task-oriented dialogue (ToD), a user holds a conversation with an artificial agent  with the aim of completing a concrete task. Although this technology represents one of  the central objectives of AI and has been the focus of ever more intense research and  development efforts, it is currently limited to a few narrow domains (e.g., food ordering,  ticket booking) and a handful of languages (e.g., English, Chinese). This work provides an  extensive overview of existing methods and resources in multilingual ToD as an entry point  to this exciting and emerging field. We find that the most critical factor preventing the  creation of truly multilingual ToD systems is the lack of datasets in most languages for  both training and evaluation. In fact, acquiring annotations or human feedback for each  component of modular systems or for data-hungry end-to-end systems is expensive and  tedious. Hence, state-of-the-art approaches to multilingual ToD mostly rely on (zero- or  few-shot) cross-lingual transfer from resource-rich languages (almost exclusively English),  either by means of (i) machine translation or (ii) multilingual representations. These  approaches are currently viable only for typologically similar languages and languages with  parallel / monolingual corpora available. On the other hand, their effectiveness beyond these  boundaries is doubtful or hard to assess due to the lack of linguistically diverse benchmarks  (especially for natural language generation and end-to-end evaluation). To overcome this  limitation, we draw parallels between components of the ToD pipeline and other NLP tasks,  which can inspire solutions for learning in low-resource scenarios. Finally, we list additional  challenges that multilinguality poses for related areas (such as speech, fluency in generated  text, and human-centred evaluation), and indicate future directions that hold promise to  further expand language coverage and dialogue capabilities of current ToD systems. 

While state-of-the-art models that rely upon massively multilingual pretrained encoders achieve sample efficiency in downstream applications, they still require abundant amounts of unlabelled text. Nevertheless, most of the world's languages lack such resources. Hence, we investigate a more radical form of unsupervised knowledge transfer in the absence of linguistic data. In particular, for the first time we pretrain neural networks via emergent communication from referential games. Our key assumption is that grounding communication on images---as a crude approximation of real-world environments---inductively biases the model towards learning natural languages. On the one hand, we show that this substantially benefits machine translation in few-shot settings. On the other hand, this also provides an extrinsic evaluation protocol to probe the properties of emergent languages ex vitro. Intuitively, the closer they are to natural languages, the higher the gains from pretraining on them should be. For instance, in this work we measure the influence of communication success and maximum sequence length on downstream performances. Finally, we introduce a customised adapter layer and annealing strategies for the regulariser of maximum-a-posteriori inference during fine-tuning. These turn out to be crucial to facilitate knowledge transfer and prevent catastrophic forgetting. Compared to a recurrent baseline, our method yields gains of $59.0\%$$\sim$$147.6\%$ in BLEU score with only $500$ NMT training instances and $65.1\%$$\sim$$196.7\%$ with $1,000$ NMT training instances across four language pairs. These proof-of-concept results reveal the potential of emergent communication pretraining for both natural language processing tasks in resource-poor settings and extrinsic evaluation of artificial languages.