Government
PAC Prediction Sets Under Covariate Shift
Park, Sangdon, Dobriban, Edgar, Lee, Insup, Bastani, Osbert
An important challenge facing modern machine learning is how to rigorously quantify the uncertainty of model predictions. Conveying uncertainty is especially important when there are changes to the underlying data distribution that might invalidate the predictive model. Yet, most existing uncertainty quantification algorithms break down in the presence of such shifts. We propose a novel approach that addresses this challenge by constructing \emph{probably approximately correct (PAC)} prediction sets in the presence of covariate shift. Our approach focuses on the setting where there is a covariate shift from the source distribution (where we have labeled training examples) to the target distribution (for which we want to quantify uncertainty). Our algorithm assumes given importance weights that encode how the probabilities of the training examples change under the covariate shift. In practice, importance weights typically need to be estimated; thus, we extend our algorithm to the setting where we are given confidence intervals for the importance weights rather than their true value. We demonstrate the effectiveness of our approach on various covariate shifts designed based on the DomainNet and ImageNet datasets.
CIRA Guide to Custom Loss Functions for Neural Networks in Environmental Sciences -- Version 1
Ebert-Uphoff, Imme, Lagerquist, Ryan, Hilburn, Kyle, Lee, Yoonjin, Haynes, Katherine, Stock, Jason, Kumler, Christina, Stewart, Jebb Q.
Neural networks are increasingly used in environmental science applications. Furthermore, neural network models are trained by minimizing a loss function, and it is crucial to choose the loss function very carefully for environmental science applications, as it determines what exactly is being optimized. Standard loss functions do not cover all the needs of the environmental sciences, which makes it important for scientists to be able to develop their own custom loss functions so that they can implement many of the classic performance measures already developed in environmental science, including measures developed for spatial model verification. However, there are very few resources available that cover the basics of custom loss function development comprehensively, and to the best of our knowledge none that focus on the needs of environmental scientists. This document seeks to fill this gap by providing a guide on how to write custom loss functions targeted toward environmental science applications. Topics include the basics of writing custom loss functions, common pitfalls, functions to use in loss functions, examples such as fractions skill score as loss function, how to incorporate physical constraints, discrete and soft discretization, and concepts such as focal, robust, and adaptive loss. While examples are currently provided in this guide for Python with Keras and the TensorFlow backend, the basic concepts also apply to other environments, such as Python with PyTorch. Similarly, while the sample loss functions provided here are from meteorology, these are just examples of how to create custom loss functions. Other fields in the environmental sciences have very similar needs for custom loss functions, e.g., for evaluating spatial forecasts effectively, and the concepts discussed here can be applied there as well. All code samples are provided in a GitHub repository.
Physion: Evaluating Physical Prediction from Vision in Humans and Machines
Bear, Daniel M., Wang, Elias, Mrowca, Damian, Binder, Felix J., Tung, Hsiau-Yu Fish, Pramod, R. T., Holdaway, Cameron, Tao, Sirui, Smith, Kevin, Sun, Fan-Yun, Fei-Fei, Li, Kanwisher, Nancy, Tenenbaum, Joshua B., Yamins, Daniel L. K., Fan, Judith E.
While machine learning algorithms excel at many challenging visual tasks, it is unclear that they can make predictions about commonplace real world physical events. Here, we present a visual and physical prediction benchmark that precisely measures this capability. In realistically simulating a wide variety of physical phenomena -- rigid and soft-body collisions, stable multi-object configurations, rolling and sliding, projectile motion -- our dataset presents a more comprehensive challenge than existing benchmarks. Moreover, we have collected human responses for our stimuli so that model predictions can be directly compared to human judgments. We compare an array of algorithms -- varying in their architecture, learning objective, input-output structure, and training data -- on their ability to make diverse physical predictions. We find that graph neural networks with access to the physical state best capture human behavior, whereas among models that receive only visual input, those with object-centric representations or pretraining do best but fall far short of human accuracy. This suggests that extracting physically meaningful representations of scenes is the main bottleneck to achieving human-like visual prediction. We thus demonstrate how our benchmark can identify areas for improvement and measure progress on this key aspect of physical understanding.
Entropy-based Logic Explanations of Neural Networks
Barbiero, Pietro, Ciravegna, Gabriele, Giannini, Francesco, Liรณ, Pietro, Gori, Marco, Melacci, Stefano
Explainable artificial intelligence has rapidly emerged since lawmakers have started requiring interpretable models for safety-critical domains. Concept-based neural networks have arisen as explainable-by-design methods as they leverage human-understandable symbols (i.e. concepts) to predict class memberships. However, most of these approaches focus on the identification of the most relevant concepts but do not provide concise, formal explanations of how such concepts are leveraged by the classifier to make predictions. In this paper, we propose a novel end-to-end differentiable approach enabling the extraction of logic explanations from neural networks using the formalism of First-Order Logic. The method relies on an entropy-based criterion which automatically identifies the most relevant concepts. We consider four different case studies to demonstrate that: (i) this entropy-based criterion enables the distillation of concise logic explanations in safety-critical domains from clinical data to computer vision; (ii) the proposed approach outperforms state-of-the-art white-box models in terms of classification accuracy.
Maxmin-Fair Ranking: Individual Fairness under Group-Fairness Constraints
Garcia-Soriano, David, Bonchi, Francesco
The bulk of the algorithmic fairness literature deals with group fairness along the lines of demographic parity [9] or equal opportunity We study a novel problem of fairness in ranking aimed at minimizing [16]: this is typically expressed by means of some fairness the amount of individual unfairness introduced when enforcing constraint requiring that the top-positions (for any) in the ranking group-fairness constraints. Our proposal is rooted in the contain enough elements from some groups that are protected distributional maxmin fairness theory, which uses randomization from discrimination based on sex, race, age, etc. In fact, [6] shows to maximize the expected satisfaction of the worst-off individuals.
The promise and perils of Artificial Intelligence partnerships
"A period that had been broadly described as engagement has come to an end," Kurt Campbell, the Indo-Pacific Coordinator at the United States (US) National Security Council, told a virtual audience in May on the subject of US-China relations. "The dominant paradigm is going to be competition." On several occasions, Campbell has highlighted that one of the major arenas of this competition will concern technology. This is increasingly reflected in US national security structures. Today, there is both a senior director and coordinator for technology and national security at the White House; the National Economic Council has briefed the Cabinet on supply chain resilience; and the focus of Department of Defense policy reviews have been on emerging military technologies.
Experts Doubt Ethical AI Design Will Be Broadly Adopted as the Norm Within the Next Decade
This is the 12th "Future of the Internet" canvassing Pew Research Center and Elon University's Imagining the Internet Center have conducted together to get expert views about important digital issues. In this case, the questions focused on the prospects for ethical artificial intelligence (AI) by the year 2030. This is a nonscientific canvassing based on a nonrandom sample; this broad array of opinions about where current trends may lead in the next decade represents only the points of view of the individuals who responded to the queries. Pew Research and Elon's Imagining the Internet Center built a database of experts to canvass from a wide range of fields, choosing to invite people from several sectors, including professionals and policy people based in government bodies, nonprofits and foundations, technology businesses, think tanks and in networks of interested academics and technology innovators. The predictions reported here came in response to a set of questions in an online canvassing conducted between June 30 and July 27, 2020. In all, 602 technology innovators and developers, business and policy leaders, researchers and activists responded to at least one of the questions covered in this report. More on the methodology underlying this canvassing and the participants can be found in the final section. Artificial intelligence systems "understand" and shape a lot of what happens in people's lives. AI applications "speak" to people and answer questions when the name of a digital voice assistant is called out. They run the chatbots that handle customer-service issues people have with companies. They help diagnose cancer and other medical conditions. They scour the use of credit cards for signs of fraud, and they determine who could be a credit risk. They help people drive from point A to point B and update traffic information to shorten travel times. They are the operating system of driverless vehicles. They sift applications to make recommendations about job candidates. They determine the material that is offered up in people's newsfeeds and video choices. They recognize people's faces, translate languages and suggest how to complete people's sentences or search queries. They can "read" people's emotions. They beat them at sophisticated games.
Council Post: Data's Double Edges: How To Use Machine Learning To Solve The Problem Of Unused Data In Risk Management
Gary M. Shiffman, Ph.D. is the Founder and CEO of Giant Oak and Co-Founder and CEO of Consilient. He is the creator of GOST and Dozer. According to my company's research, a full 25% of PPP fraud cases brought by the Department of Justice could have been easily prevented. The fraud is so obviously clumsy that it is embarrassing to whomever approved the loans. Decision-makers consume a lot of data.
AI breast cancer screening project wins government funding for NHS trial
UK researchers have secured government funding to study the use of artificial intelligence for breast cancer screening in NHS hospitals. The work builds on previous research which showed that artificial intelligence could be as effective as human radiologists in spotting breast cancer from X-ray images. Backed by funding through the Artificial Intelligence in Health and Care Award, the next stages of the project aim to further assess the feasibility of the AI system to see how the technology could be integrated into the national screening programme in the future to support clinicians. The partnership, which includes Imperial College London, Google Health, Imperial College Healthcare NHS Trust, St George's Hospitals NHS Foundation Trust, and the Royal Surrey NHS Foundation Trust builds on previous work, in which the researchers trained the algorithm on depersonalised patient data and mammograms from patients in the UK and US. The findings, published in Nature in January 2020, showed the AI system was able to correctly identify cancers from the images with a similar degree of accuracy to expert radiologists, and demonstrated potential to assist clinical staff in practice.
Artificial Intelligence Services Bangalore
Hyena, the best Artificial Intelligence services provider in Bangalore, India, delivers next-generation Artificial Intelligence services and solutions for enterprises. We believe that AI will alter the way enterprises perform to grasp customer attention in this competitive world. To cope with this trend, by leveraging the power of cognitive automation and data analytics, we build next-level AI solutions that illuminate new growth opportunities and operational models for enterprises.