This thesis investigates how natural language understanding and generation with transformer models can benefit from grounding the models with knowledge representations. Currently, the most prevailing paradigm for training language models is through pre-training on abundant raw text data and fine-tuning on downstream tasks. Although language models continue to advance, especially the recent trend of Large Language Models (LLMs) such as ChatGPT, there seem to be limits to what can be achieved with text data alone and it is desirable to study the impact of applying and integrating rich forms of knowledge representation to improve model performance. The most widely used form of knowledge for language modelling is structured knowledge in the form of triples consisting of entities and their relationships, often in English. This thesis explores beyond this conventional approach and aims to address several key questions: Can knowledge of entities extend its benefits beyond entity-centric tasks such as entity linking? How can we faithfully and effectively extract such structured knowledge from raw text, especially noisy web text? How do other types of knowledge, beyond structured knowledge, contribute to improving NLP tasks?

Online updating of time series forecasting models aims to tackle the challenge of concept drifting by adjusting forecasting models based on streaming data. While numerous algorithms have been developed, most of them focus on model design and updating. In practice, many of these methods struggle with continuous performance regression in the face of accumulated concept drifts over time. To address this limitation, we present a novel approach, Concept \textbf{D}rift \textbf{D}etection an\textbf{D} \textbf{A}daptation (D3A), that first detects drifting conception and then aggressively adapts the current model to the drifted concepts after the detection for rapid adaption. To best harness the utility of historical data for model adaptation, we propose a data augmentation strategy introducing Gaussian noise into existing training instances. It helps mitigate the data distribution gap, a critical factor contributing to train-test performance inconsistency. The significance of our data augmentation process is verified by our theoretical analysis. Our empirical studies across six datasets demonstrate the effectiveness of D3A in improving model adaptation capability. Notably, compared to a simple Temporal Convolutional Network (TCN) baseline, D3A reduces the average Mean Squared Error (MSE) by $43.9\%$. For the state-of-the-art (SOTA) model, the MSE is reduced by $33.3\%$.

Emerging technologies like generative AI tools, including ChatGPT, are increasingly utilized in educational settings, offering innovative approaches to learning while simultaneously posing new challenges. This study employs a survey methodology to examine the policy landscape concerning these technologies, drawing insights from 102 high school principals and higher education provosts. Our results reveal a prominent policy gap: the majority of institutions lack specialized guide-lines for the ethical deployment of AI tools such as ChatGPT. Moreover,we observed that high schools are less inclined to work on policies than higher educational institutions. Where such policies do exist, they often overlook crucial issues, including student privacy and algorithmic transparency. Administrators overwhelmingly recognize the necessity of these policies, primarily to safeguard student safety and mitigate plagiarism risks. Our findings underscore the urgent need for flexible and iterative policy frameworks in educational contexts.

Algorithmic decision-making is increasingly being adopted across public higher education. The expansion of data-driven practices by post-secondary institutions has occurred in parallel with the adoption of New Public Management approaches by neoliberal administrations. In this study, we conduct a qualitative analysis of an in-depth ethnographic case study of data and algorithms in use at a public college in Ontario, Canada. We identify the data, algorithms, and outcomes in use at the college. We assess how the college's processes and relationships support those outcomes and the different stakeholders' perceptions of the college's data-driven systems. In addition, we find that the growing reliance on algorithmic decisions leads to increased student surveillance, exacerbation of existing inequities, and the automation of the faculty-student relationship. Finally, we identify a cycle of increased institutional power perpetuated by algorithmic decision-making, and driven by a push towards financial sustainability.

In early 2021, the writer Lucy Sante sent an e-mail to her closest friends. Its subject was "A Bombshell," which Sante later joked was an unintentional pun. In the text she attached, she explained that at the age of sixty-six she was accepting her long-suppressed identity as a transgender woman. Her transition had been catalyzed by an interaction with artificial intelligence. That February, Sante had downloaded FaceApp, a photo-editing application that uses neural networks to generate realistic transformations of people's faces.

A Florida Christian school teacher was arrested this week after allegedly creating child sexual abuse materials using photos from the school yearbook and artificial intelligence (AI), according to authorities. The Pasco County Sheriff'sOffice said 67-year-old Steven Houser of New Port Richey faces charges for possession of child pornography. Deputies initiated an investigation after receiving an unspecified tip about Houser. Steven Guy Houser, a third-grade science teacher at a Christian school in New Port Richey, Florida, was allegedly found to be in possession of child pornography he created using yearbook photos and artificial intelligence. The investigation discovered that Beacon, a third-grade science teacher at Beacon Christian Academy, allegedly possessed two photos and three videos depicting child pornography.

Alberto M. Carvalho sought to establish the nation's second-largest school system as a hub for innovation. "This is a technology that becomes a personal assistant to students," Carvalho said at a demonstration at Roybal Leaning Center, west of downtown. "It demystifies the navigation of the day ... crunches the data in a way that it brings what students need." Other school systems have pioneered similar platforms with chatbots -- as educators try to better engage students and families to improve learning and attendance. But Ed takes such efforts to the highest level yet, said Joanna Smith-Griffin, chief executive of Boston-based AllHere, which has worked with districts across the country and won a 6-million contract over five years to guide the L.A. effort. At its core, Ed is designed to give students immediate answers about where they stand, what they need to do to make progress -- or, more immediately, find out when their bus will arrive.

In recent times, the grandeur of Large Language Models (LLMs) has not only shone in the realm of natural language processing but has also cast its brilliance across a vast array of applications. This remarkable display of LLM capabilities has ignited a surge in research contributions within this domain, spanning a diverse spectrum of topics. These contributions encompass advancements in neural network architecture, context length enhancements, model alignment, training datasets, benchmarking, efficiency improvements, and more. Recent years have witnessed a dynamic synergy between academia and industry, propelling the field of LLM research to new heights. A notable milestone in this journey is the introduction of ChatGPT, a powerful AI chatbot grounded in LLMs, which has garnered widespread societal attention. The evolving technology of LLMs has begun to reshape the landscape of the entire AI community, promising a revolutionary shift in the way we create and employ AI algorithms. Given this swift-paced technical evolution, our survey embarks on a journey to encapsulate the recent strides made in the world of LLMs. Through an exploration of the background, key discoveries, and prevailing methodologies, we offer an up-to-the-minute review of the literature. By examining multiple LLM models, our paper not only presents a comprehensive overview but also charts a course that identifies existing challenges and points toward potential future research trajectories.

In federated learning, data heterogeneity significantly impacts performance. A typical solution involves segregating these parameters into shared and personalized components, a concept also relevant in multi-task learning. Addressing this, we propose "Loop Improvement" (LI), a novel method enhancing this separation and feature extraction without necessitating a central server or data interchange among participants. Our experiments reveal LI's superiority in several aspects: In personalized federated learning environments, LI consistently outperforms the advanced FedALA algorithm in accuracy across diverse scenarios. Additionally, LI's feature extractor closely matches the performance achieved when aggregating data from all clients. In global model contexts, employing LI with stacked personalized layers and an additional network also yields comparable results to combined client data scenarios. Furthermore, LI's adaptability extends to multi-task learning, streamlining the extraction of common features across tasks and obviating the need for simultaneous training. This approach not only enhances individual task performance but also achieves accuracy levels on par with classic multi-task learning methods where all tasks are trained simultaneously. LI integrates a loop topology with layer-wise and end-to-end training, compatible with various neural network models. This paper also delves into the theoretical underpinnings of LI's effectiveness, offering insights into its potential applications. The code is on https://github.com/axedge1983/LI

The growing interest in Large Language Models (LLMs) and in particular in conversational models with which users can interact has led to the development of a large number of open-source chat LLMs. These models are evaluated on a wide range of benchmarks to assess their capabilities in answering questions or solving problems on almost any possible topic or to test their ability to reason or interpret texts. Instead, the evaluation of the knowledge that these models have of the languages has received much less attention. For example, the words that they can recognize and use in different languages. In this paper, we evaluate the knowledge that open-source chat LLMs have of Spanish words by testing a sample of words in a reference dictionary. The results show that open-source chat LLMs produce incorrect meanings for an important fraction of the words and are not able to use most of the words correctly to write sentences with context. These results show how Spanish is left behind in the open-source LLM race and highlight the need to push for linguistic fairness in conversational LLMs ensuring that they provide similar performance across languages.