The fields of AI current lacks methods to quantitatively assess and potentially alter the moral values inherent in the output of large language models (LLMs). However, decades of social science research has developed and refined widely-accepted moral value surveys, such as the World Values Survey (WVS), eliciting value judgments from direct questions in various geographies. We have turned those questions into value statements and use NLP to compute to how well popular LLMs are aligned with moral values for various demographics and cultures. While the WVS is accepted as an explicit assessment of values, we lack methods for assessing implicit moral and cultural values in media, e.g., encountered in social media, political rhetoric, narratives, and generated by AI systems such as LLMs that are increasingly present in our daily lives. As we consume online content and utilize LLM outputs, we might ask, which moral values are being implicitly promoted or undercut, or -- in the case of LLMs -- if they are intending to represent a cultural identity, are they doing so consistently? In this paper we utilize a Recognizing Value Resonance (RVR) NLP model to identify WVS values that resonate and conflict with a given passage of output text. We apply RVR to the text generated by LLMs to characterize implicit moral values, allowing us to quantify the moral/cultural distance between LLMs and various demographics that have been surveyed using the WVS. In line with other work we find that LLMs exhibit several Western-centric value biases; they overestimate how conservative people in non-Western countries are, they are less accurate in representing gender for non-Western countries, and portray older populations as having more traditional values. Our results highlight value misalignment and age groups, and a need for social science informed technological solutions addressing value plurality in LLMs.

Analytic software tools and workflows are increasing in capability, complexity, number, and scale, and the integrity of our workflows is as important as ever. Specifically, we must be able to inspect the process of analytic workflows to assess (1) confidence of the conclusions, (2) risks and biases of the operations involved, (3) sensitivity of the conclusions to sources and agents, (4) impact and pertinence of various sources and agents, and (5) diversity of the sources that support the conclusions. We present an approach that tracks agents' provenance with PROV-O in conjunction with agents' appraisals and evidence links (expressed in our novel DIVE ontology). Together, PROV-O and DIVE enable dynamic propagation of confidence and counter-factual refutation to improve human-machine trust and analytic integrity. We demonstrate representative software developed for user interaction with that provenance, and discuss key needs for organizations adopting such approaches. We demonstrate all of these assessments in a multi-agent analysis scenario, using an interactive web-based information validation UI.

Many modern machine learning approaches require vast amounts of training data to learn new concepts; conversely, human learning often requires few examples--sometimes only one--from which the learner can abstract structural concepts. We present a novel approach to introducing new spatial structures to an AI agent, combining deep learning over qualitative spatial relations with various heuristic search algorithms. The agent extracts spatial relations from a sparse set of noisy examples of block-based structures, and trains convolutional and sequential models of those relation sets. To create novel examples of similar structures, the agent begins placing blocks on a virtual table, uses a CNN to predict the most similar complete example structure after each placement, an LSTM to predict the most likely set of remaining moves needed to complete it, and recommends one using heuristic search. We verify that the agent learned the concept by observing its virtual block-building activities, wherein it ranks each potential subsequent action toward building its learned concept. We empirically assess this approach with human participants' ratings of the block structures. Initial results and qualitative evaluations of structures generated by the trained agent show where it has generalized concepts from the training data, which heuristics perform best within the search space, and how we might improve learning and execution.

The 28th International Workshop on Qualitative Reasoning (QR-15) presented advances toward reasoning tractably with massive qualitative and quantitative models, automatically learning and reasoning about continuous processes, and representing knowledge about space, causation, and uncertainty.

When human experts build qualitative or quantitative models of complex systems, they use the function of the system as a guideline to decide what to model and how to model it, yet they do not often encode this functional knowledge directly. If qualitative and quantitative models contained this functional knowledge, our reasoning systems might use it as a heuristic or as a filter during the course of quantitative and qualitative simulation. Matthew Klenk (PARC) delivered a separate talk related to massive-scale model-based reasoning, describing the challenge of choosing initial conditions for simulation. Throughout the technical presentations on advances in qualitative simulation, we discussed the practicality of automatically transforming quantitative and qualitative models during the course of reasoning.

Reasoning with commonsense science knowledge is an important challenge for Artificial Intelligence. This paper presents a system that revises its knowledge in a commonsense science domain by constructing and evaluating explanations. Domain knowledge is represented using qualitative model fragments, which are used to explain phenomena via model formulation. Metareasoning is used to (1) score competing explanations numerically along several dimensions and (2) evaluate preferred explanations for global consistency. Inconsistencies cause the system to favor alternative explanations and thereby change its beliefs. We simulate the belief changes of several students during clinical interviews about how the seasons change. We show that qualitative models accurately represent student knowledge and that our system produces and revises a sequence of explanations similar those of the students.

Understanding conceptual change is an important problem in modeling human cognition and in making integrated AI systems that can learn autonomously. This paper describes a model of explanation-based conceptual change, integrating sketch understanding, analogical processing, qualitative models, truth-maintenance, and heuristic-based reasoning within the Companions cognitive architecture. Sketch understanding is used to automatically encode stimuli in the form of comic strips. Qualitative models and conceptual quantities are constructed for new phenomena via analogical reasoning and heuristics. Truth-maintenance is used to integrate conceptual and episodic knowledge into explanations, and heuristics are used to modify existing conceptual knowledge in order to produce better explanations. We simulate the learning and revision of the concept of force, testing the concepts learned via a questionnaire of sketches given to students, showing that our model follows a similar learning trajectory.