Currently, image-text-driven multi-modal deep learning models have demonstrated their outstanding potential in many fields. In practice, tasks centered around facial images have broad application prospects. This paper presents \textbf{FaceCaption-15M}, a large-scale, diverse, and high-quality dataset of facial images accompanied by their natural language descriptions (facial image-to-text). This dataset aims to facilitate a study on face-centered tasks. FaceCaption-15M comprises over 15 million pairs of facial images and their corresponding natural language descriptions of facial features, making it the largest facial image-caption dataset to date. We conducted a comprehensive analysis of image quality, text naturalness, text complexity, and text-image relevance to demonstrate the superiority of FaceCaption-15M. To validate the effectiveness of FaceCaption-15M, we first trained a facial language-image pre-training model (FLIP, similar to CLIP) to align facial image with its corresponding captions in feature space. Subsequently, using both image and text encoders and fine-tuning only the linear layer, our FLIP-based models achieved state-of-the-art results on two challenging face-centered tasks. The purpose is to promote research in the field of face-related tasks through the availability of the proposed FaceCaption-15M dataset. All data, codes, and models are publicly available. https://huggingface.co/datasets/OpenFace-CQUPT/FaceCaption-15M

Evaluation is critical for assessing capabilities, tracking scientific progress, and informing model selection. In this paper, we present three desiderata for a good benchmark for language models: (i) salience (e.g., knowledge about World War II is more salient than a random day in history), (ii) novelty (i.e., the benchmark reveals new trends in model rankings not shown by previous benchmarks), and (iii) difficulty (i.e., the benchmark should be difficult for existing models, leaving headroom for future improvement). We operationalize these three desiderata and cast benchmark creation as a search problem, that of finding benchmarks that that satisfy all three desiderata. To tackle this search problem, we present AutoBencher, which uses a language model to automatically search for datasets that meet the three desiderata. AutoBencher uses privileged information (e.g. relevant documents) to construct reliable datasets, and adaptivity with reranking to optimize for the search objective. We use AutoBencher to create datasets for math, multilingual, and knowledge-intensive question answering. The scalability of AutoBencher allows it to test fine-grained categories and tail knowledge, creating datasets that are on average 27% more novel and 22% more difficult than existing benchmarks. A closer investigation of our constructed datasets shows that we can identify specific gaps in LM knowledge in language models that are not captured by existing benchmarks, such as Gemini Pro performing much worse on question answering about the Permian Extinction and Fordism, while OpenAGI-7B performing surprisingly well on QA about COVID-19.

Pre-trained large language models (LLMs) can now be easily adapted for specific business purposes using custom prompts or fine tuning. These customizations are often iteratively re-engineered to improve some aspect of performance, but after each change businesses want to ensure that there has been no negative impact on the system's behavior around such critical issues as bias. Prior methods of benchmarking bias use techniques such as word masking and multiple choice questions to assess bias at scale, but these do not capture all of the nuanced types of bias that can occur in free response answers, the types of answers typically generated by LLM systems. In this paper, we identify several kinds of nuanced bias in free text that cannot be similarly identified by multiple choice tests. We describe these as: confidence bias, implied bias, inclusion bias and erasure bias. We present a semi-automated pipeline for detecting these types of bias by first eliminating answers that can be automatically classified as unbiased and then co-evaluating name reversed pairs using crowd workers. We believe that the nuanced classifications our method generates can be used to give better feedback to LLMs, especially as LLM reasoning capabilities become more advanced.

Deep neural networks and other modern machine learning models are often susceptible to adversarial attacks. Indeed, an adversary may often be able to change a model's prediction through a small, directed perturbation of the model's input - an issue in safety-critical applications. Adversarially robust machine learning is usually based on a minmax optimisation problem that minimises the machine learning loss under maximisation-based adversarial attacks. In this work, we study adversaries that determine their attack using a Bayesian statistical approach rather than maximisation. The resulting Bayesian adversarial robustness problem is a relaxation of the usual minmax problem. To solve this problem, we propose Abram - a continuous-time particle system that shall approximate the gradient flow corresponding to the underlying learning problem. We show that Abram approximates a McKean-Vlasov process and justify the use of Abram by giving assumptions under which the McKean-Vlasov process finds the minimiser of the Bayesian adversarial robustness problem. We discuss two ways to discretise Abram and show its suitability in benchmark adversarial deep learning experiments.

A pair of lion brothers have made the longest swim ever recorded for their species – about 1.5 kilometres across hippo and crocodile-infested waters. The massive swim – equivalent to the aquatic leg of an Olympic triathlon – was the pair's fourth attempt to cross the Kazinga Channel in Queen Elizabeth National Park, Uganda, and was recorded by a drone-mounted thermal camera at night. The lions had to abort earlier attempts after encountering large animals, most likely hippos or Nile crocodiles, which are also visible in the footage. Making the effort even more extraordinary, one of the lions, named Jacob, has only three legs. Jacob has had an extremely challenging life, says Alexander Braczkowski at Griffith University in Australia: he has been gored by a buffalo, his family was poisoned for the lion body-part trade, he was caught in a poacher's snare and he eventually lost his leg after it was stuck in a poacher's steel trap.

The rapid advancement of large language models (LLMs) has paved the way for the development of highly capable autonomous agents. However, existing multi-agent frameworks often struggle with integrating diverse capable third-party agents due to reliance on agents defined within their own ecosystems. They also face challenges in simulating distributed environments, as most frameworks are limited to single-device setups. Furthermore, these frameworks often rely on hard-coded communication pipelines, limiting their adaptability to dynamic task requirements. Inspired by the concept of the Internet, we propose the Internet of Agents (IoA), a novel framework that addresses these limitations by providing a flexible and scalable platform for LLM-based multi-agent collaboration. IoA introduces an agent integration protocol, an instant-messaging-like architecture design, and dynamic mechanisms for agent teaming and conversation flow control. Through extensive experiments on general assistant tasks, embodied AI tasks, and retrieval-augmented generation benchmarks, we demonstrate that IoA consistently outperforms state-of-the-art baselines, showcasing its ability to facilitate effective collaboration among heterogeneous agents. IoA represents a step towards linking diverse agents in an Internet-like environment, where agents can seamlessly collaborate to achieve greater intelligence and capabilities. Our codebase has been released at \url{https://github.com/OpenBMB/IoA}.

Using 286 research papers collected from Web of Science, ScienceDirect, SpringerLink, arXiv, and Google Scholar databases, a systematic review methodology was adopted to review and summarize the current challenges and potential future developments in data, algorithms, and evaluation aspects of RSs. It was found that RSs involve five major research topics, namely algorithmic improvement, domain applications, user behavior & cognition, data processing & modeling, and social impact & ethics. Collaborative filtering and hybrid recommendation techniques are mainstream. The performance of RSs is jointly limited by four types of eight data issues, two types of twelve algorithmic issues, and two evaluation issues. Notably, data-related issues such as cold start, data sparsity, and data poisoning, algorithmic issues like interest drift, device-cloud collaboration, non-causal driven, and multitask conflicts, along with evaluation issues such as offline data leakage and multi-objective balancing, have prominent impacts. Fusing physiological signals for multimodal modeling, defending against data poisoning through user information behavior, evaluating generative recommendations via social experiments, fine-tuning pre-trained large models to schedule device-cloud resource, enhancing causal inference with deep reinforcement learning, training multi-task models based on probability distributions, using cross-temporal dataset partitioning, and evaluating recommendation objectives across the full lifecycle are feasible solutions to address the aforementioned prominent challenges and unlock the power and value of RSs.The collected literature is mainly based on major international databases, and future research will further expand upon it.

This thesis investigates the application of state-of-the-art advances in generative neural networks for fast simulation of the Zero Degree Calorimeter (ZDC) neutron detector in the ALICE experiment at CERN. Traditional simulation methods using the GEANT Monte Carlo toolkit, while accurate, are computationally demanding. With increasing computational needs at CERN, efficient simulation techniques are essential. The thesis provides a comprehensive literature review on the application of neural networks in computer vision, fast simulations using machine learning, and generative neural networks in high-energy physics. The theory of the analyzed models is also discussed, along with technical aspects and the challenges associated with a practical implementation. The experiments evaluate various neural network architectures, including convolutional neural networks, vision transformers, and MLP-Mixers, as well as generative frameworks such as autoencoders, generative adversarial networks, vector quantization models, and diffusion models. Key contributions include the implementation and evaluation of these models, a significant improvement in the Wasserstein metric compared to existing methods with a low generation time of 5 milliseconds per sample, and the formulation of a list of recommendations for developing models for fast ZDC simulation. Open-source code and detailed hyperparameter settings are provided for reproducibility. Additionally, the thesis outlines future research directions to further enhance simulation fidelity and efficiency.

The potential for untrustworthy conversational agents presents a significant threat for covert social manipulation. Taking inspiration from Frank Herbert's "Dune", where the Bene Gesserit Sisterhood uses the Voice for influence, manipulation, and control of people, we explore how generative AI provides a way to implement individualized influence at industrial scales. Already, these models can manipulate communication across text, image, speech, and most recently video. They are rapidly becoming affordable enough for any organization of even moderate means to train and deploy. If employed by malicious actors, they risk becoming powerful tools for shaping public opinion, sowing discord, and undermining organizations from companies to governments. As researchers and developers, it is crucial to recognize the potential for such weaponization and to explore strategies for prevention, detection, and defense against these emerging forms of sociotechnical manipulation.

In this work, we combine the benefits of both ideas and demonstrate an efficient dynamic encoder training framework. Varying-size models are often required to deploy ASR systems We leverage score-based layer-wise pruning to find the optimal under different hardware and/or application constraints such layer combination for the subnets, saving the computationally as memory and latency. To avoid redundant training and optimization expensive search required by the general supernet training efforts for individual models of different sizes, we methods [9, 10]. Furthermore, we design an efficient two-step present the dynamic encoder size approach, which jointly trains training pipeline. In Step 1, we propose two methods, Simple-multiple performant models within one supernet from scratch. Top-k and Iterative-Zero-Out, to effectively learn the associated These subnets of various sizes are layer-wise pruned from the layer importance scores in a data-driven way. In step 2, we generate supernet, and thus, enjoy full parameter sharing. By combining binary masks for all subnets and exploit the sandwich rule score-based pruning with supernet training, we propose two [6] for efficient joint training of the supernet and subnets. Additionally, novel methods, Simple-Top-k and Iterative-Zero-Out, to automatically we explore different training techniques to mitigate select the best-performing subnets in a data-driven the mutual training inference and further boost the word error manner, avoiding resource-intensive search efforts.