Larger language models have higher accuracy on average, but are they better on every single instance (datapoint)? Some work suggests larger models have higher out-of-distribution robustness, while other work suggests they have lower accuracy on rare subgroups. To understand these differences, we investigate these models at the level of individual instances. However, one major challenge is that individual predictions are highly sensitive to noise in the randomness in training. We develop statistically rigorous methods to address this, and after accounting for pretraining and finetuning noise, we find that our BERT-Large is worse than BERT-Mini on at least 1-4% of instances across MNLI, SST-2, and QQP, compared to the overall accuracy improvement of 2-10%. We also find that finetuning noise increases with model size and that instance-level accuracy has momentum: improvement from BERT-Mini to BERT-Medium correlates with improvement from BERT-Medium to BERT-Large. Our findings suggest that instance-level predictions provide a rich source of information; we therefore, recommend that researchers supplement model weights with model predictions.

Abstract-- Applications that need to sense, measure, and gather real-time information from the environment frequently face three main restrictions: power consumption, cost, and lack of infrastructure. Most of the challenges imposed by these limitations can be better addressed by embedding Machine Learning (ML) classifiers in the hardware that senses the environment, creating smart sensors able to interpret the low-level data stream. However, for this approach to be cost-effective, we need highly efficient classifiers suitable to execute in unresourceful hardware, such as low-power microcontrollers. In this paper, we present an open-source tool named EmbML - Embedded Machine Learning that implements a pipeline to develop classifiers for resource-constrained hardware. We describe its implementation details and provide a comprehensive analysis of its classifiers considering accuracy, classification time, and memory usage. Moreover, we compare the performance of its classifiers with classifiers produced by related tools to demonstrate that our tool provides a diverse set of classification algorithms that are both compact and accurate. Therefore, these smart sensors are more powerefficient since they eliminate the need for communicating all the raw data. PPLICATIONS that need to sense, measure, and gather real-time information from the environment frequently of interest - e.g., a dry soil crop area that needs watering or face three main restrictions [1]: power consumption, cost, the capture of a disease-vector mosquito.

While AI has benefited humans, it may also harm humans if not appropriately developed. We conducted a literature review of current related work in developing AI systems from an HCI perspective. Different from other approaches, our focus is on the unique characteristics of AI technology and the differences between non-AI computing systems and AI systems. We further elaborate on the human-centered AI (HCAI) approach that we proposed in 2019. Our review and analysis highlight unique issues in developing AI systems which HCI professionals have not encountered in non-AI computing systems. To further enable the implementation of HCAI, we promote the research and application of human-AI interaction (HAII) as an interdisciplinary collaboration. There are many opportunities for HCI professionals to play a key role to make unique contributions to the main HAII areas as we identified. To support future HCI practice in the HAII area, we also offer enhanced HCI methods and strategic recommendations. In conclusion, we believe that promoting the HAII research and application will further enable the implementation of HCAI, enabling HCI professionals to address the unique issues of AI systems and develop human-centered AI systems.

Defeasible reasoning is the mode of reasoning where conclusions can be overturned by taking into account new evidence. A commonly used method in philosophy and AI literature is to handcraft argumentation supporting inference graphs. While humans find inference graphs very useful for reasoning, constructing them at scale is difficult. In this paper, we automatically generate such inference graphs through transfer learning from another NLP task that shares the kind of reasoning that inference graphs support. Through automated metrics and human evaluation, we find that our method generates meaningful graphs for the defeasible inference task. Human accuracy on this task improves by 20% by consulting the generated graphs. Our findings open up exciting new research avenues for cases where machine reasoning can help human reasoning. (A dataset of 230,000 influence graphs for each defeasible query is located at: https://tinyurl.com/defeasiblegraphs.)

We present an approach, based on learning an intrinsic data manifold, for the initialization of the internal state values of LSTM recurrent neural networks, ensuring consistency with the initial observed input data. Exploiting the generalized synchronization concept, we argue that the converged, "mature" internal states constitute a function on this learned manifold. The dimension of this manifold then dictates the length of observed input time series data required for consistent initialization. We illustrate our approach through a partially observed chemical model system, where initializing the internal LSTM states in this fashion yields visibly improved performance. Finally, we show that learning this data manifold enables the transformation of partially observed dynamics into fully observed ones, facilitating alternative identification paths for nonlinear dynamical systems.

The answer: both suffer from a "truthiness" problem. Truthiness is a term coined by Stephen Colbert to describe the tactic of weaving facts into a false narrative. Conspiracy theories like QAnon rely on truthiness, using individual data points to reach wild and untrue conclusions like ISIS was created by the CIA or a hidden Deep State runs the U.S. government. Like it or not, human beings are highly susceptible to truthiness: research indicates that 50% of Americans believe in at least one conspiracy theory. The AI business also suffers from a "truthy" belief: that when black box algorithms are used to make high-stakes decisions--like who gets approved for a loan, a job interview, or even an organ transplant--the fact that we don't know HOW these algorithms reach their decisions is not a problem so long as an AI developer can "explain" a model's reasoning.

This analysis summarizes and reflects on the following research: Suchman, L. (2020). Project Maven was introduced to the media and U.S. general public in the summer of 2018 when several Google employees voiced concerns over the company's contract to automate the labeling of images from U.S. military drones to determine "objects of interest" (including vehicles, buildings, and persons) with sparse details on its intended purpose. In solidarity, academic researchers supported Google employees' concerns and added that "further automation…of the US drone program can only serve to worsen an operation that is already highly problematic, even arguably illegal and immoral under the laws and norms of armed conflict." While Google decided not to renew the contract following its employees' protests, the project's contract was picked up and continued by a different company. This story, and Project Maven more generally, typifies an intersection of critical security studies and technology studies that Lucy Suchman examines in research on automation and artificial intelligence technologies in U.S. counterterrorism strategies.

NATO officials are kicking around a new set of questions for member states on artificial intelligence in defense applications, as the alliance seeks common ground ahead of a strategy document planned for this summer. The move comes amid a grand effort to sharpen NATO's edge in what officials call emerging and disruptive technologies, or EDT. Autonomous and artificial intelligence-enabled weaponry is a key element in that push, aimed at ensuring tech leadership on a global scale. Exactly where the alliance falls on the spectrum between permitting AI-powered defense technology in some applications and disavowing it in others is expected to be a hotly debated topic in the run-up to the June 14 NATO summit. "We have agreed that we need principles of responsible use, but we're also in the process of delineating specific technologies," David van Weel, the alliance's assistant secretary-general for emerging security challenges, said at a web event earlier this month organized by the Estonian Defence Ministry.

It's now a given that countries worldwide are battling for AI supremacy. To date, most of the public discussion surrounding this competition has focused on commercial gains flowing from the technology. But the AI arms race for military applications is racing ahead as well, and concerned scientists, academics, and AI industry leaders have been sounding the alarm. Compared to existing military capabilities, AI-enabled technology can make decisions on the battlefield with mathematical speed and accuracy and never get tired. However, countries and organizations developing this tech are only just beginning to articulate ideas about how ethics will influence the wars of the near future.

Half a year since launch, and it's still a challenge for a lot of us to get our hands on a next-gen console. Sony has reportedly said there will be supply constraints for the rest of the year, but we didn't even need to be told. If you've tried to buy an Xbox Series X or PS5, you've probably had to shop around (online, mostly) for stock. What have you missed out on? According to Jessica Conditt, our newest consoles are still finding their feet, with Sony and Microsoft's differing approaches still yet to yield anything conclusive.