This paper aims to introduce and analyze the Viz system in a comprehensive way, a novel system architecture that integrates Quantized Low-Rank Adapters (QLoRA) to fine-tune large language models (LLM) within a legally compliant and resource efficient marketplace. Viz represents a significant contribution to the field of artificial intelligence, particularly in addressing the challenges of computational efficiency, legal compliance, and economic sustainability in the utilization and monetization of LLMs. The paper delineates the scholarly discourse and developments that have informed the creation of Viz, focusing primarily on the advancements in LLM models, copyright issues in AI training (NYT case, 2023), and the evolution of model fine-tuning techniques, particularly low-rank adapters and quantized low-rank adapters, to create a sustainable and economically compliant framework for LLM utilization. The economic model it proposes benefits content creators, AI developers, and end-users, delineating a harmonious integration of technology, economy, and law, offering a comprehensive solution to the complex challenges of today's AI landscape.

Large language models (LLMs) [1-5] have recently become the new darling of academia and industry due to their remarkable performance in a variety of natural language processing (NLP) tasks. With the blessing of techniques such as large-scale pre-training [6], instruction tuning [7], and reinforcement learning from human feedback (RLHF) [8, 9], existing pretrained LLMs have demonstrated unique capabilities in language understanding, generation, interaction, and reasoning. These powerful capabilities of LLMs also drive many emergent research topics (e.g., instruction learning [10], in-context learning [1], chain-of-thought prompting [11], etc.) to further investigate their huge potentials, and bring unlimited possibilities for humans to build advanced artificial intelligence systems. However, alongside these advancements, a pressing issue that plagues LLMs has been widely criticized as "hallucination", referred to as a phenomenon where LLMs tend to generate text that is incorrect, nonsensical, or not real [12]. To alleviate the phenomenon of "hallucination" during the generation of LLMs, a promising direction is to retrieve the factual knowledge that are highly relevant to the user query, and then guide LLMs to generate response according to the retrieved context, resulting in retrieval-augmented LMs [13, 14] that have recently demonstrated strong performance in knowledge intensive tasks, especially for knowledge-based question answering (KBQA). The workflow of existing retrieval-augmented LMs [15, 16] mainly relies on embedding-based retrieval methods, which will first encode various forms of knowledge base and also the user query into the same latent space, then use a semantic similarity metric to retrieve the top-K most relevant documents as prompt, and finally instruct LLMs to only use the provided context to answer the user query.

Multi-modal intent detection aims to utilize various modalities to understand the user's intentions, which is essential for the deployment of dialogue systems in real-world scenarios. The two core challenges for multi-modal intent detection are (1) how to effectively align and fuse different features of modalities and (2) the limited labeled multi-modal intent training data. In this work, we introduce a shallow-to-deep interaction framework with data augmentation (SDIF-DA) to address the above challenges. Firstly, SDIF-DA leverages a shallow-to-deep interaction module to progressively and effectively align and fuse features across text, video, and audio modalities. Secondly, we propose a ChatGPT-based data augmentation approach to automatically augment sufficient training data. Experimental results demonstrate that SDIF-DA can effectively align and fuse multi-modal features by achieving state-of-the-art performance. In addition, extensive analyses show that the introduced data augmentation approach can successfully distill knowledge from the large language model.

Query-focused summarization (QFS) aims to provide a summary of a single document/multi documents that can satisfy the information needs of a given query. It is useful for various real-world applications, such as abstractive snippet generation or more recent retrieval augmented generation (RAG). A prototypical QFS pipeline consists of a retriever (sparse or dense retrieval) and a generator (usually a large language model). However, applying large language models (LLM) potentially leads to hallucinations, especially when the evidence contradicts the prior belief of LLMs. There has been growing interest in developing new decoding methods to improve generation quality and reduce hallucination. In this work, we conduct a large-scale reproducibility study on one recently proposed decoding method -- Context-aware Decoding (CAD). In addition to replicating CAD's experiments on news summarization datasets, we include experiments on QFS datasets, and conduct more rigorous analysis on computational complexity and hyperparameter sensitivity. Experiments with eight different language models show that performance-wise, CAD improves QFS quality by (1) reducing factuality errors/hallucinations while (2) mostly retaining the match of lexical patterns, measured by ROUGE scores, while also at a cost of increased inference-time FLOPs and reduced decoding speed. The code implementation based on Huggingface Library is made available https://github.com/zhichaoxu-shufe/context-aware-decoding-qfs

The scale and quality of point cloud datasets constrain the advancement of point cloud learning. Recently, with the development of multi-modal learning, the incorporation of domain-agnostic prior knowledge from other modalities, such as images and text, to assist in point cloud feature learning has been considered a promising avenue. Existing methods have demonstrated the effectiveness of multi-modal contrastive training and feature distillation on point clouds. However, challenges remain, including the requirement for paired triplet data, redundancy and ambiguity in supervised features, and the disruption of the original priors. In this paper, we propose a language-assisted approach to point cloud feature learning (LAST-PCL), enriching semantic concepts through LLMs-based text enrichment. We achieve de-redundancy and feature dimensionality reduction without compromising textual priors by statistical-based and training-free significant feature selection. Furthermore, we also delve into an in-depth analysis of the impact of text contrastive training on the point cloud. Extensive experiments validate that the proposed method learns semantically meaningful point cloud features and achieves state-of-the-art or comparable performance in 3D semantic segmentation, 3D object detection, and 3D scene classification tasks.

To understand how models resolve such conflicts, we study Llama-2-chat models on the forbidden fact task. Specifically, we instruct Llama-2 to truthfully complete a factual recall statement while forbidding it from saying the correct answer. This often makes the model give incorrect answers. We decompose Llama-2 into 1000+ components, and rank each one with respect to how useful it is for forbidding the correct answer. We find that in aggregate, around 35 components are enough to reliably implement the full suppression behavior. However, these components are fairly heterogeneous and many operate using faulty heuristics. We discover that one of these heuristics can be exploited via a manually designed adversarial attack which we call The California Attack. Our results highlight some roadblocks standing in the way of being able to successfully interpret advanced ML systems.

The Italian Digital Media Observatory (IDMO) project, part of a European initiative, focuses on countering disinformation and fake news. This report outlines contributions from Rai-CRITS to the project, including: (i) the creation of novel datasets for testing technologies (ii) development of an automatic model for categorizing Pagella Politica verdicts to facilitate broader analysis (iii) creation of an automatic model for recognizing textual entailment with exceptional accuracy on the FEVER dataset (iv) assessment using GPT-4 to detecting content treatment style (v) a game to raise awareness about fake news at national events.

Causal inference is one of the hallmarks of human intelligence. While the field of CausalNLP has attracted much interest in the recent years, existing causal inference datasets in NLP primarily rely on discovering causality from empirical knowledge (e.g., commonsense knowledge). In this work, we propose the first benchmark dataset to test the pure causal inference skills of large language models (LLMs). Specifically, we formulate a novel task Corr2Cause, which takes a set of correlational statements and determines the causal relationship between the variables. We curate a large-scale dataset of more than 200K samples, on which we evaluate seventeen existing LLMs. Through our experiments, we identify a key shortcoming of LLMs in terms of their causal inference skills, and show that these models achieve almost close to random performance on the task. This shortcoming is somewhat mitigated when we try to re-purpose LLMs for this skill via finetuning, but we find that these models still fail to generalize -- they can only perform causal inference in in-distribution settings when variable names and textual expressions used in the queries are similar to those in the training set, but fail in out-of-distribution settings generated by perturbing these queries. Corr2Cause is a challenging task for LLMs, and would be helpful in guiding future research on improving LLMs' pure reasoning skills and generalizability. Our data is at https://huggingface.co/datasets/causalnlp/corr2cause. Our code is at https://github.com/causalNLP/corr2cause.

A few days ago, Microsoft released a standalone Android app for Microsoft Copilot, giving you a quick way to access the AI assistant. Turns out the iOS and iPad versions weren't far behind, because they're now available from Apple's App Store. Just like in Copilot on desktop and other AI chatbots, such as ChatGPT, you can type in your question and wait for responses generated by artificial intelligence. In Copilot's case, you'll get responses spun by OpenAI's GPT-4, the company's latest large language model. The free version of ChatGPT, in comparison, is powered by the older ChatGPT-3.5, and you'll need to pay for ChatGPT Plus to get access to the newer model.

The advancement of Large Language Models (LLMs) has opened up new avenues for tackling annotation tasks in the field of social sciences. These models, especially the newer iterations like Chat-GPT or GPT-4, are now being used to annotate textual data (Gilardi, Alizadeh, & Kubli, 2023; Heseltine & Hohenberg, 2023; Møller, Dalsgaard, Pera, & Aiello, 2023; Ziems et al., 2023), which can be helpful for analyzing various social and political phenomena (Törnberg, 2023; Ziems et al., 2023). However, a significant challenge arises when there is a necessity to share research data with proprietary and closed models that are provided by companies due to the utilization of APIs (Ollion, Shen, Macanovic, & Chatelain, 2023; Spirling, 2023). This is particularly concerning in scenarios where data sharing is not preferable due to data privacy. In light of this, open models which can be operated on independent devices like university servers, present a viable alternative (Alizadeh et al., 2023). They allow researchers to harness the potential of generative large language models without compromising data security. This paper endeavors to promote the adoption of open models by providing two examples and guidelines for leveraging them instead of proprietary models for annotation tasks within the social sciences.