Mark Zuckerberg is poised for a packed week in Asia, with stops in Japan, South Korea and India to work on artificial intelligence services and forge ties with business and government leaders in the region. Meta Platforms' CEO, who's been in Japan the past week to accompany his family on a ski trip, will meet with his company's developers in Tokyo on Monday and Tuesday, according to a spokesperson. They'll discuss the company's Quest virtual-reality headsets and its large language model Llama, one of its efforts to compete with OpenAI's ChatGPT. On Wednesday, Zuckerberg plans to travel to Seoul to meet with partners, the spokesperson said, declining to elaborate further. He is tentatively scheduled to meet with South Korean President Yoon Suk Yeol, according to a person familiar with the matter who asked not to be named because the details are private.

Semantic Overlap Summarization (SOS) is a constrained multi-document summarization task, where the constraint is to capture the common/overlapping information between two alternative narratives. While recent advancements in Large Language Models (LLMs) have achieved superior performance in numerous summarization tasks, a benchmarking study of the SOS task using LLMs is yet to be performed. As LLMs' responses are sensitive to slight variations in prompt design, a major challenge in conducting such a benchmarking study is to systematically explore a variety of prompts before drawing a reliable conclusion. Fortunately, very recently, the TELeR taxonomy has been proposed which can be used to design and explore various prompts for LLMs. Using this TELeR taxonomy and 15 popular LLMs, this paper comprehensively evaluates LLMs on the SOS Task, assessing their ability to summarize overlapping information from multiple alternative narratives. For evaluation, we report well-established metrics like ROUGE, BERTscore, and SEM-F1$ on two different datasets of alternative narratives. We conclude the paper by analyzing the strengths and limitations of various LLMs in terms of their capabilities in capturing overlapping information The code and datasets used to conduct this study are available at https://anonymous.4open.science/r/llm_eval-E16D.

Federated Instruction Tuning (FIT) has shown the ability to achieve collaborative model instruction tuning among massive data owners without sharing private data. However, it still faces two key challenges, i.e., data and resource heterogeneity. Due to the varying data distribution and preferences among data owners, FIT cannot adapt to the personalized data of individual owners. Moreover, clients with superior computational abilities are constrained since they need to maintain the same fine-tuning architecture as the weaker clients. To address these issues, we propose a novel Personalized Federated Instruction Tuning (PerFIT) framework based on architecture search. Specifically, PerFIT allows each client to search for a personalized architecture by expanding the trainable parameter space of the global model followed by pruning the parameters to the original state. This procedure allows personalized instruction fine-tuning within expanded parameter spaces, concurrently preserving the same number of trainable parameters. Furthermore, to release the abilities of heterogeneous computational resources and enhance the performance of personalization on local data, we exploit personalized parameter-wise aggregation. The evaluation with multiple LLMs non-IID scenarios demonstrates that compared to the state-of-the-art FIT methods, our approach can achieve up to a 23% decrease in perplexity.

Machine unlearning can be useful for removing harmful capabilities and memorized text from large language models (LLMs), but there are not yet standardized methods for rigorously evaluating it. Second, we apply a comprehensive set of tests for the robustness and competitiveness of unlearning in the "Who's Harry Potter" (WHP) model from Eldan and Russinovich (2023). While WHP's unlearning generalizes well when evaluated with the "Familiarity" metric from Eldan and Russinovich, we find i) higher-than-baseline amounts of knowledge can reliably be extracted, ii) WHP performs on par with the original model on Harry Potter Q&A tasks, iii) it represents latent knowledge comparably to the original model, and iv) there is collateral unlearning in related domains. Overall, our results highlight the importance of comprehensive unlearning evaluation that avoids ad-hoc metrics. It is difficult to ensure that large language models (LLMs) will always behave harmlessly. Meanwhile, LLMs also memorize pretraining data, raising concerns involving privacy and fair use (Carlini et al., 2022; Shi et al., 2023; Karamolegkou et al., 2023). To reduce these risks, machine unlearning has emerged as a way to remove undesirable knowledge from LLMs (Bourtoule et al., 2021; Nguyen et al., 2022; Si et al., 2023; Shaik et al., 2023; Liu et al., 2024a). Ideally, LLM unlearning should produce a model that is competitive on most tasks but which robustly loses knowledge on the unlearning task in a way that is resistant to extraction by an adversary. Prior works have introduced various ad hoc techniques (see Table 1 and Section 2). However, to date, little has been done to comprehensively evaluate LLM unlearning (Liu et al., 2024a).

The Natural Language Interface to Databases (NLIDB) empowers non-technical users with database access through intuitive natural language (NL) interactions. Advanced approaches, utilizing neural sequence-to-sequence models or large-scale language models, typically employ auto-regressive decoding to generate unique SQL queries sequentially. While these translation models have greatly improved the overall translation accuracy, surpassing 70% on NLIDB benchmarks, the use of auto-regressive decoding to generate single SQL queries may result in sub-optimal outputs, potentially leading to erroneous translations. In this paper, we propose Metasql, a unified generate-then-rank framework that can be flexibly incorporated with existing NLIDBs to consistently improve their translation accuracy. Metasql introduces query metadata to control the generation of better SQL query candidates and uses learning-to-rank algorithms to retrieve globally optimized queries. Specifically, Metasql first breaks down the meaning of the given NL query into a set of possible query metadata, representing the basic concepts of the semantics. These metadata are then used as language constraints to steer the underlying translation model toward generating a set of candidate SQL queries. Finally, Metasql ranks the candidates to identify the best matching one for the given NL query. Extensive experiments are performed to study Metasql on two public NLIDB benchmarks. The results show that the performance of the translation models can be effectively improved using Metasql.

Although our method extends the training The focus of this work is on the low-rank adapter (Hu et al., samples required for convergence by 40%, we can fit models 2022, LoRA), a subclass of linear adapters. The linearity that are 3 bigger with roughly half the bandwidth. LoRA is frequently used for finetuning transformers, often resulting in less than Our work explores a new territory in the pre-training 10% of the total trainable parameters (even as low as 0.5%). Although the forward pass incurs an extra computational overhead, the significance of LoRA parameterization pertains ReLoRA (Lialin et al., 2023) sequentially trains and merges to the optimizer memory footprint. However, it AdamW (Kingma & Ba, 2015; Loshchilov & Hutter, 2019) does not yield comparable pre-training performance without typically maintain two states for each parameter, resulting in initial full-parameter training. In contrast, our work uses memory consumption that is twice the size of the trainable parallel updates to match pre-training performance without parameters. FedLoRA focuses on the distributed 2023) achieves further memory savings by storing W in finetuning of LoRA parameters. These works have catalyzed AdaMix (Wang et al., 2022) averages all MLPs in a Mixture the development of several repositories (Wang, 2023; of Experts (MOE) into a single MLP. AdaMix requires Dettmers et al., 2023; Dettmers, 2023; huggingface, 2023), constant synchronization during the forward and backward enabling finetuning of models with billions of parameters passes. Our work requires no synchronization in the forward on low-memory devices. For an in-depth discussion of related works, see Section 5. To understand the conditions required to pre-train a model with LoRA, we first identify a specific scenario where standard Unless stated otherwise, we denote x as a scalar, x a vector, This serves as a guide for developing our algorithm that X a matrix, X a distribution or a set, f() a function and retains the memory efficiency of LoRA. Although low-rank adapters (LoRAs) have proven to be an effective finetuning method, they have apparent limitations 2.1. As evidenced in Figure 2, models parameterized with LoRA demonstrate inferior performance Adapters serve as trainable functions that modify existing compared to models trained using standard optimization. They facilitate parameterefficient This performance gap isn't surprising as it can be attributed finetuning of large-scale models by minimizing the to the inherent rank constraint in LoRA.

Approaches to aligning large language models (LLMs) with human values has focused on correcting misalignment that emerges from pretraining. However, this focus overlooks another source of misalignment: bad actors might purposely fine-tune LLMs to achieve harmful goals. In this paper, we present an emerging threat model that has arisen from alignment circumvention and fine-tuning attacks. However, lacking in previous works is a clear presentation of the conditions for effective defence. We propose a set of conditions for effective defence against harmful fine-tuning in LLMs called "Immunization conditions," which help us understand how we would construct and measure future defences. Using this formal framework for defence, we offer a synthesis of different research directions that might be persued to prevent harmful fine-tuning attacks and provide a demonstration of how to use these conditions experimentally showing early results of using an adversarial loss to immunize LLama2-7b-chat.

Temporal knowledge graph question answering (TKGQA) poses a significant challenge task, due to the temporal constraints hidden in questions and the answers sought from dynamic structured knowledge. Although large language models (LLMs) have made considerable progress in their reasoning ability over structured data, their application to the TKGQA task is a relatively unexplored area. This paper first proposes a novel generative temporal knowledge graph question answering framework, GenTKGQA, which guides LLMs to answer temporal questions through two phases: Subgraph Retrieval and Answer Generation. First, we exploit LLM's intrinsic knowledge to mine temporal constraints and structural links in the questions without extra training, thus narrowing down the subgraph search space in both temporal and structural dimensions. Next, we design virtual knowledge indicators to fuse the graph neural network signals of the subgraph and the text representations of the LLM in a non-shallow way, which helps the open-source LLM deeply understand the temporal order and structural dependencies among the retrieved facts through instruction tuning. Experimental results demonstrate that our model outperforms state-of-the-art baselines, even achieving 100\% on the metrics for the simple question type.

Large Language Models (LLMs) are reshaping the research landscape in artificial intelligence, particularly as model parameters scale up significantly, unlocking remarkable capabilities across various domains. Nevertheless, the scalability of model parameters faces constraints due to limitations in GPU memory and computational speed. To address these constraints, various weight compression methods have emerged, such as Pruning and Quantization. Given the low-rank nature of weight matrices in language models, the reduction of weights through matrix decomposition undoubtedly holds significant potential and promise. In this paper, drawing upon the intrinsic structure of LLMs, we propose a novel approach termed Data-free Joint Rank-k Approximation for compressing the parameter matrices. Significantly, our method is characterized by without necessitating additional involvement of any corpus, while simultaneously preserving orthogonality in conjunction with pruning and quantization methods. We achieve a model pruning of 80% parameters while retaining 93.43% of the original performance without any calibration data. Additionally, we explore the fundamental properties of the weight matrix of LLMs undergone Rank-k Approximation and conduct comprehensive experiments to elucidate our hypothesis.

To address the challenges of out-of-control generation in generative models for machine reading comprehension (MRC), we introduce the Question-Attended Span Extraction (QASE) module. Integrated during the fine-tuning of pre-trained generative language models (PLMs), QASE enables these PLMs to match SOTA extractive methods and outperform leading LLMs like GPT-4 in MRC tasks, without significant increases in computational costs.