Government
Robust Immersive Telepresence and Mobile Telemanipulation: NimbRo wins ANA Avatar XPRIZE Finals
Schwarz, Max, Lenz, Christian, Memmesheimer, Raphael, Pätzold, Bastian, Rochow, Andre, Schreiber, Michael, Behnke, Sven
Abstract-- Robotic avatar systems promise to bridge distances and reduce the need for travel. We present the updated NimbRo avatar system, winner of the $5M grand prize at the international ANA Avatar XPRIZE competition, which required participants to build intuitive and immersive robotic telepresence systems that could be operated by briefly trained operators. Video and audio data are compressed using low-latency HEVC and Opus codecs. We propose a new locomotion control device with tunable resistance force. To increase flexibility, the robot's upper-body height can be adjusted by the operator. Top left: Operator judge controlling the avatar. Bottom left: VR view (cropped). Reducing the need In this paper, we present and discuss the updates and to travel is thus beneficial for many reasons. While voice extensions of the NimbRo avatar system (Figure 1) that we calls and video conferencing help, they cannot replace inperson made for our highly successful participation in the ANA meetings entirely due to lack of immersion and Avatar XPRIZE Finals in November 2022, where our team social interaction.
BiasTestGPT: Using ChatGPT for Social Bias Testing of Language Models
Kocielnik, Rafal, Prabhumoye, Shrimai, Zhang, Vivian, Jiang, Roy, Alvarez, R. Michael, Anandkumar, Anima
Pretrained Language Models (PLMs) harbor inherent social biases that can result in harmful real-world implications. Such social biases are measured through the probability values that PLMs output for different social groups and attributes appearing in a set of test sentences. However, bias testing is currently cumbersome since the test sentences are generated either from a limited set of manual templates or need expensive crowd-sourcing. We instead propose using ChatGPT for the controllable generation of test sentences, given any arbitrary user-specified combination of social groups and attributes appearing in the test sentences. When compared to template-based methods, our approach using ChatGPT for test sentence generation is superior in detecting social bias, especially in challenging settings such as intersectional biases. We present an open-source comprehensive bias testing framework (BiasTestGPT), hosted on HuggingFace, that can be plugged into any open-source PLM for bias testing. User testing with domain experts from various fields has shown their interest in being able to test modern AI for social biases. Our tool has significantly improved their awareness of such biases in PLMs, proving to be learnable and user-friendly. We thus enable seamless open-ended social bias testing of PLMs by domain experts through an automatic large-scale generation of diverse test sentences for any combination of social categories and attributes.
Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition
Dooley, Samuel, Sukthanker, Rhea Sanjay, Dickerson, John P., White, Colin, Hutter, Frank, Goldblum, Micah
Face recognition systems are widely deployed in safety-critical applications, including law enforcement, yet they exhibit bias across a range of socio-demographic dimensions, such as gender and race. Conventional wisdom dictates that model biases arise from biased training data. As a consequence, previous works on bias mitigation largely focused on pre-processing the training data, adding penalties to prevent bias from effecting the model during training, or post-processing predictions to debias them, yet these approaches have shown limited success on hard problems such as face recognition. In our work, we discover that biases are actually inherent to neural network architectures themselves. Following this reframing, we conduct the first neural architecture search for fairness, jointly with a search for hyperparameters. Our search outputs a suite of models which Pareto-dominate all other high-performance architectures and existing bias mitigation methods in terms of accuracy and fairness, often by large margins, on the two most widely used datasets for face identification, CelebA and VGGFace2. Furthermore, these models generalize to other datasets and sensitive attributes. We release our code, models and raw data files at https://github.com/dooleys/FR-NAS.
A Comprehensive Review of Visual-Textual Sentiment Analysis from Social Media Networks
Al-Tameemi, Israa Khalaf Salman, Feizi-Derakhshi, Mohammad-Reza, Pashazadeh, Saeed, Asadpour, Mohammad
Social media networks have become a significant aspect of people's lives, serving as a platform for their ideas, opinions and emotions. Consequently, automated sentiment analysis (SA) is critical for recognising people's feelings in ways that other information sources cannot. The analysis of these feelings revealed various applications, including brand evaluations, YouTube film reviews and healthcare applications. As social media continues to develop, people post a massive amount of information in different forms, including text, photos, audio and video. Thus, traditional SA algorithms have become limited, as they do not consider the expressiveness of other modalities. By including such characteristics from various material sources, these multimodal data streams provide new opportunities for optimising the expected results beyond text-based SA. Our study focuses on the forefront field of multimodal SA, which examines visual and textual data posted on social media networks. Many people are more likely to utilise this information to express themselves on these platforms. To serve as a resource for academics in this rapidly growing field, we introduce a comprehensive overview of textual and visual SA, including data pre-processing, feature extraction techniques, sentiment benchmark datasets, and the efficacy of multiple classification methodologies suited to each field. We also provide a brief introduction of the most frequently utilised data fusion strategies and a summary of existing research on visual-textual SA. Finally, we highlight the most significant challenges and investigate several important sentiment applications.
Hidden yet quantifiable: A lower bound for confounding strength using randomized trials
De Bartolomeis, Piersilvio, Abad, Javier, Donhauser, Konstantin, Yang, Fanny
In the era of fast-paced precision medicine, observational studies play a major role in properly evaluating new treatments in clinical practice. Yet, unobserved confounding can significantly compromise causal conclusions drawn from non-randomized data. We propose a novel strategy that leverages randomized trials to quantify unobserved confounding. First, we design a statistical test to detect unobserved confounding with strength above a given threshold. Then, we use the test to estimate an asymptotically valid lower bound on the unobserved confounding strength. We evaluate the power and validity of our statistical test on several synthetic and semi-synthetic datasets. Further, we show how our lower bound can correctly identify the absence and presence of unobserved confounding in a real-world setting.
Multi-Group Fairness Evaluation via Conditional Value-at-Risk Testing
Paes, Lucas Monteiro, Suresh, Ananda Theertha, Beutel, Alex, Calmon, Flavio P., Beirami, Ahmad
Machine learning (ML) models used in prediction and classification tasks may display performance disparities across population groups determined by sensitive attributes (e.g., race, sex, age). We consider the problem of evaluating the performance of a fixed ML model across population groups defined by multiple sensitive attributes (e.g., race and sex and age). To address this issue, we propose an approach to test for performance disparities based on Conditional Value-at-Risk (CVaR). By allowing a small probabilistic slack on the groups over which a model has approximately equal performance, we show that the sample complexity required for discovering performance violations is reduced exponentially to be at most upper bounded by the square root of the number of groups. As a byproduct of our analysis, when the groups are weighted by a specific prior distribution, we show that Rényi entropy of order 2/3 of the prior distribution captures the sample complexity of the proposed CVaR test algorithm. Finally, we also show that there exists a non-i.i.d. Machine learning (ML) algorithms are increasingly used in domains of consequence such as hiring [1], lending [2], [3], policing [4], and healthcare [5], [6].
ExpM+NF Tractable Exponential Mechanism via Normalizing Flow, A Path through the Accuracy-Privacy Ceiling Constraining Differentially Private ML
Bridges, Robert A., Tombs, Vandy J., Stanley, Christopher B.
The Exponential Mechanism (ExpM), a differentially private optimization method, promises many advantages over Differentially Private Stochastic Gradient Descent (DPSGD), the state-of-the-art (SOTA) and de facto method for differentially private machine learning (ML). Yet, ExpM has been historically stymied from differentially private training of modern ML algorithms by two obstructions: ExpM requires a sensitivity bound for the given loss function; ExpM requires sampling from a historically intractable density. We prove a sensitivity bound for $\ell(2)$ loss, and investigate using Normalizing Flows (NFs), deep networks furnishing approximate sampling from the otherwise intractable ExpM distribution. We prove that as the NF output converges to ExpM distribution, the privacy ($\varepsilon$) of an NF sample converges to that of the ExpM distribution. Under the assumption that the NF output distribution is the ExpM distribution, we empirically test ExpM+NF against DPSGD using the SOTA implementation (Opacus \cite{opacus} with PRV accounting) in multiple classification tasks on the Adult Dataset (census data) and MIMIC-III Dataset (healthcare records) using Logistic Regression and GRU-D, a deep learning recurrent neural network with \smallsim 20K-100K parameters. In all experiments we find ExpM+NF achieves greater than 94\% of the non-private training accuracy (AUC) with $\varepsilon$-DP for $\varepsilon$ a low as $1\mathrm{e}{-3}$ -- three orders of magnitude stronger privacy with similar accuracy. Further, performance results show ExpM+NF training time is comparable to (slightly less) than DPSGD. Limitations and future directions are provided; notably, research on NF approximation accuracy and its effect on privacy are a promising avenue to substantially advancing the field. Code for these experiments \hl{will be provided after review}.
Causal Estimation of Exposure Shifts with Neural Networks: Evaluating the Health Benefits of Stricter Air Quality Standards in the US
Tec, Mauricio, Mudele, Oladimeji, Josey, Kevin, Dominici, Francesca
In policy research, one of the most critical analytic tasks is to estimate the causal effect of a policy-relevant shift to the distribution of a continuous exposure/treatment on an outcome of interest. We call this problem shift-response function (SRF) estimation. Existing neural network methods involving robust causal-effect estimators lack theoretical guarantees and practical implementations for SRF estimation. Motivated by a key policy-relevant question in public health, we develop a neural network method and its theoretical underpinnings to estimate SRFs with robustness and efficiency guarantees. We then apply our method to data consisting of 68 million individuals and 27 million deaths across the U.S. to estimate the causal effect from revising the US National Ambient Air Quality Standards (NAAQS) for PM 2.5 from 12 $\mu g/m^3$ to 9 $\mu g/m^3$. This change has been recently proposed by the US Environmental Protection Agency (EPA). Our goal is to estimate, for the first time, the reduction in deaths that would result from this anticipated revision using causal methods for SRFs. Our proposed method, called {T}argeted {R}egularization for {E}xposure {S}hifts with Neural {Net}works (TRESNET), contributes to the neural network literature for causal inference in two ways: first, it proposes a targeted regularization loss with theoretical properties that ensure double robustness and achieves asymptotic efficiency specific for SRF estimation; second, it enables loss functions from the exponential family of distributions to accommodate non-continuous outcome distributions (such as hospitalization or mortality counts). We complement our application with benchmark experiments that demonstrate TRESNET's broad applicability and competitiveness.
Meta and IBM launch 'AI Alliance' to promote open-source AI development
Facebook parent Meta and IBM on Tuesday launched a new group called the AI Alliance advocating for an "open-science" approach to AI development that puts them at odds with rivals Google, Microsoft and ChatGPT-maker OpenAI. These two diverging camps – the open and the closed – disagree about whether to build AI in a way that makes the underlying technology widely accessible. Safety is at the heart of the debate, but so is who gets to profit from AI's advances. Open advocates favor an approach that is "not proprietary and closed", said Darío Gil, a senior vice-president at IBM who directs its research division. "So it's not like a thing that is locked in a barrel and no one knows what they are."