Information access systems, such as search engines, recommender systems, and conversational assistants, have become integral to our daily lives as they help us satisfy our information needs. However, evaluating the effectiveness of these systems presents a long-standing and complex scientific challenge. This challenge is rooted in the difficulty of assessing a system's overall effectiveness in assisting users to complete tasks through interactive support, and further exacerbated by the substantial variation in user behaviour and preferences. To address this challenge, user simulation emerges as a promising solution. This book focuses on providing a thorough understanding of user simulation techniques designed specifically for evaluation purposes. We begin with a background of information access system evaluation and explore the diverse applications of user simulation. Subsequently, we systematically review the major research progress in user simulation, covering both general frameworks for designing user simulators, utilizing user simulation for evaluation, and specific models and algorithms for simulating user interactions with search engines, recommender systems, and conversational assistants. Realizing that user simulation is an interdisciplinary research topic, whenever possible, we attempt to establish connections with related fields, including machine learning, dialogue systems, user modeling, and economics. We end the book with a detailed discussion of important future research directions, many of which extend beyond the evaluation of information access systems and are expected to have broader impact on how to evaluate interactive intelligent systems in general.

Urban air pollution has been linked to various human health concerns, including cardiopulmonary diseases. Communities who suffer from poor air quality often rely on experts to identify pollution sources due to the lack of accessible tools. Taking this into account, we developed Smell Pittsburgh, a system that enables community members to report odors and track where these odors are frequently concentrated. All smell report data are publicly accessible online. These reports are also sent to the local health department and visualized on a map along with air quality data from monitoring stations. This visualization provides a comprehensive overview of the local pollution landscape. Additionally, with these reports and air quality data, we developed a model to predict upcoming smell events and send push notifications to inform communities. We also applied regression analysis to identify statistically significant effects of push notifications on user engagement. Our evaluation of this system demonstrates that engaging residents in documenting their experiences with pollution odors can help identify local air pollution patterns, and can empower communities to advocate for better air quality. All citizen-contributed smell data are publicly accessible and can be downloaded from https://smellpgh.org.

Most of the existing recommender systems use the ratings provided by users on individual items. An additional source of preference information is to use the ratings that users provide on sets of items. The advantages of using preferences on sets are two-fold. First, a rating provided on a set conveys some preference information about each of the set's items, which allows us to acquire a user's preferences for more items that the number of ratings that the user provided. Second, due to privacy concerns, users may not be willing to reveal their preferences on individual items explicitly but may be willing to provide a single rating to a set of items, since it provides some level of information hiding. This paper investigates two questions related to using set-level ratings in recommender systems. First, how users' item-level ratings relate to their set-level ratings. Second, how collaborative filtering-based models for item-level rating prediction can take advantage of such set-level ratings. We have collected set-level ratings from active users of Movielens on sets of movies that they have rated in the past. Our analysis of these ratings shows that though the majority of the users provide the average of the ratings on a set's constituent items as the rating on the set, there exists a significant number of users that tend to consistently either under- or over-rate the sets. We have developed collaborative filtering-based methods to explicitly model these user behaviors that can be used to recommend items to users. Experiments on real data and on synthetic data that resembles the under- or over-rating behavior in the real data, demonstrate that these models can recover the overall characteristics of the underlying data and predict the user's ratings on individual items.

Many interactive intelligent systems, such as recommendation and information retrieval systems, treat users as a passive data source. Yet, users form mental models of systems and instead of passively providing feedback to the queries of the system, they will strategically plan their actions within the constraints of the mental model to steer the system and achieve their goals faster. We propose to explicitly account for the user's theory of the AI's mind in the user model: the intelligent system has a model of the user having a model of the intelligent system. We study a case where the system is a contextual bandit and the user model is a Markov decision process that plans based on a simpler model of the bandit. Inference in the model can be reduced to probabilistic inverse reinforcement learning, with the nested bandit model defining the transition dynamics, and is implemented using probabilistic programming. Our results show that improved performance is achieved if users can form accurate mental models that the system can capture, implying predictability of the interactive intelligent system is important not only for the user experience but also for the design of the system's statistical models.

The field of human-computer interaction (HCI) offers designers and developers of interactive systems a large repertoire of methods for ensuring that their systems will be both usable and useful. This article offers a brief introduction to these methods, focusing on the ways in which they may need to be adapted and extended to take into account the characteristic properties of systems that include some sort of AI. The discussion is organized around three types of activity: understanding users' needs, interaction design, and evaluation. The application of these methods is described with terms such as usability engineering and user-centered design. These activities constitute part of the overall software development process, and they must be carefully coordinated with the other activities involved in that process.

The inclusion of AI in an interactive system brings many potential benefits, but there are also potential side effects for the usability of the systems that ought to be taken into account from the very start. This article offers a systematic method for analyzing such usability side effects, the goal being to provide a solid basis for decisions about how to avoid or mitigate them. The analysis schema is applied in turn to nine classes of side effect. Many ideas that have been discussed in earlier literature are synthesized within this framework, which also brings into focus some concepts and issues that have received little attention so far. The section "Taming the Savage Beast of HCI" in the theme article by Lieberman (2009) offers a recent perspective.

When creating algorithms or systems that are supposed to be used by people, we should be able to adopt a "binocular" view of users' interaction with intelligent systems: a view that regards the design of interaction and the design of intelligent algorithms as interrelated parts of a single design problem. This special issue offers a coherent set of articles on two levels of generality that illustrate the binocular view and help readers to adopt it. You're more likely to hear it when listening to folks who are interested in the human side of computer use-- such as people in the field of human-computer interaction (HCI). But how much distance is there between these two fields? After all, the algorithms developed in AI research are often intended to be deployed in systems that involve some sort of interaction with users.

There is considerable validity to this point of view: Anyone who develops systems that are intended for use by people can benefit from familiarity with and application of these methods. Accordingly, this article offers a brief introduction to these methods, including examples and suggestions for additional reading (see in particular the Further Reading section). Even people who are already experts in the application of these methods should be aware of potential adaptations and extensions to the methods when applied to systems that are designed to incorporate significant use of AI. The theme articles by Lieberman (2009) and by Jameson (2009) in this issue discuss some of the ways in which systems that incorporate intelligence tend to differ from systems that do not, both in terms of their potential to help users and in terms of possible side effects. These and other properties of intelligent systems can affect the application of design and evaluation methods in various ways, some of which are illustrated in the case studies of this special issue. To organize our discussion, we distinguish broadly three types of activity that are involved in usability engineering: understanding users' needs, interaction design, and evaluation. Except for the fact that understanding users' needs tends to occur early in the design process, these activities generally proceed in parallel and complement each other.

These unintended negative consequences of the introduction of intelligence often have no direct relationship with the intended benefits, just as the adverse effects of a medication may bear no obvious relationship to the intended benefits of taking that medicine. Therefore, these negative consequences can be seen as side effects. The purpose of this article is to give designers, developers, and users of interactive intelligent systems a detailed awareness of the potential side effects of AI. As with medications, awareness of the side effects can have different implications: We may be relieved to see that a given side effect is unlikely to occur in our particular case. We may become convinced that it will inevitably occur and therefore decide not to "take the medicine" (that is, decide to stick with mainstream systems). Or most likely and most constructively, by looking carefully at the causes of the side effects and the conditions under which they can occur, we can figure out how to exploit the benefits of AI in interactive systems while avoiding the side effects.