Task-oriented dialogue systems are broadly used in virtual assistants and other automated services, providing interfaces between users and machines to facilitate specific tasks. Nowadays, task-oriented dialogue systems have greatly benefited from pre-trained language models (PLMs). However, their task-solving performance is constrained by the inherent capacities of PLMs, and scaling these models is expensive and complex as the model size becomes larger. To address these challenges, we propose Soft Mixture-of-Expert Task-Oriented Dialogue system (SMETOD) which leverages an ensemble of Mixture-of-Experts (MoEs) to excel at subproblems and generate specialized outputs for task-oriented dialogues. SMETOD also scales up a task-oriented dialogue system with simplicity and flexibility while maintaining inference efficiency. We extensively evaluate our model on three benchmark functionalities: intent prediction, dialogue state tracking, and dialogue response generation. Experimental results demonstrate that SMETOD achieves state-of-the-art performance on most evaluated metrics. Moreover, comparisons against existing strong baselines show that SMETOD has a great advantage in the cost of inference and correctness in problem-solving.

Tracking dialogue states is an essential topic in task-oriented dialogue systems, which involve filling in the necessary information in pre-defined slots corresponding to a schema. While general pre-trained language models have been shown effective in slot-filling, their performance is limited when applied to specific domains. We propose a graph-based framework that learns domain-specific prompts by incorporating the dialogue schema. Specifically, we embed domain-specific schema encoded by a graph neural network into the pre-trained language model, which allows for relations in the schema to guide the model for better adaptation to the specific domain. Our experiments demonstrate that the proposed graph-based method outperforms other multi-domain DST approaches while using similar or fewer trainable parameters. We also conduct a comprehensive study of schema graph architectures, parameter usage, and module ablation that demonstrate the effectiveness of our model on multi-domain dialogue state tracking.

Recent advances in cross-lingual commonsense reasoning (CSR) are facilitated by the development of multilingual pre-trained models (mPTMs). While mPTMs show the potential to encode commonsense knowledge for different languages, transferring commonsense knowledge learned in large-scale English corpus to other languages is challenging. To address this problem, we propose the attention-based Cross-LIngual Commonsense Knowledge transfER (CLICKER) framework, which minimizes the performance gaps between English and non-English languages in commonsense question-answering tasks. CLICKER effectively improves commonsense reasoning for non-English languages by differentiating non-commonsense knowledge from commonsense knowledge. Experimental results on public benchmarks demonstrate that CLICKER achieves remarkable improvements in the cross-lingual CSR task for languages other than English.

Nowadays, the requirements of deploying an increasing number of services across a variety of domains raise challenges With the demanding need for deploying dialogue systems in to DST models in production [4]. However, existing new domains with less cost, zero-shot dialogue state tracking dialogue datasets only span a few domains, making it impossible (DST), which tracks user's requirements in task-oriented dialogues to train a DST model upon all conceivable conversation without training on desired domains, draws attention flows [5]. Furthermore, dialogue systems are required to infer increasingly. Although prior works have leveraged questionanswering dialogue states with dynamic techniques and offer diverse (QA) data to reduce the need for in-domain training interfaces for different services. Despite the fact that the copy in DST, they fail to explicitly model knowledge transfer mechanism [6] or dialogue acts [7] are leveraged to efficiently and fusion for tracking dialogue states. To address this issue, track slots and values in the dialogue history, the performance we propose CoFunDST, which is trained on domain-agnostic of DST still relies on a large number of annotations of dialogue QA datasets and directly uses candidate choices of slot-values states, which is expensive and inefficient to collect data as knowledge for zero-shot dialogue-state generation, based for every new domain and service.