Regulation of Multi-Agent Systems (MAS) was a research topic of the past decade and one of these proposals was Electronic Institutions. However, with the recent reformulation of Artificial Neural Networks (ANN) as Deep Learning (DL), Security, Privacy, Ethical and Legal issues regarding the use of DL has raised concerns in the Artificial Intelligence (AI) Community. Now that the Regulation of MAS is almost correctly addressed, we propose the Regulation of ANN as Agent-based Training of a special type of regulated ANN that we call Institutional Neural Network. This paper introduces the former concept and provides $\mathcal{I}$, a language previously used to model and extend Electronic Institutions, as a means to implement and regulate DL.

There is a growing interest in Multi-Agent Reinforcement Learning (MARL) as the first steps towards building general intelligent agents that learn to make low and high-level decisions in non-stationary complex environments in the presence of other agents. Previous results point us towards increased conditions for coordination, efficiency/fairness, and common-pool resource sharing. We further study coordination in multi-task environments where several rewarding tasks can be performed and thus agents don't necessarily need to perform well in all tasks, but under certain conditions may specialize. An observation derived from the study is that epsilon greedy exploration of value-based reinforcement learning methods is not adequate for multi-agent independent learners because the epsilon parameter that controls the probability of selecting a random action synchronizes the agents artificially and forces them to have deterministic policies at the same time. By using policy-based methods with independent entropy regularised exploration updates, we achieved a better and smoother convergence. Another result that needs to be further investigated is that with an increased number of agents specialization tends to be more probable.

L OSS AVERSION FOSTERS COORDINATION AMONG INDEPENDENT REINFORCEMENT LEARNERS ARX IV VERSION Marco Jerome Gasparrini University Pompeu Fabra Barcelona, Spain Martí Sánchez-Fibla † University Pompeu Fabra Barcelona, Spain January 1, 2020 A BSTRACT We study what are the factors that can accelerate the emergence of collaborative behaviours among independent selfish learning agents. We depart from the "Battle of the Exes" (BoE), a spatial repeated game from which human behavioral data has been obtained (by Hawkings and Goldstone, 2016) that we find interesting because it considers two cases: a classic game theory version, called ballistic, in which agents can only make one action/decision (equivalent to the Battle of the Sexes) and a spatial version, called dynamic, in which agents can change decision (a spatial continuous version). We model both versions of the game with independent reinforcement learning agents and we manipulate the reward function transforming it into an utility introducing "loss aversion": the reward that an agent obtains can be perceived as less valuable when compared to what the other got. We prove experimentally the introduction of loss aversion fosters cooperation by accelerating its appearance, and by making it possible in some cases like in the dynamic condition. We suggest that this may be an important factor explaining the rapid converge of human behaviour towards collaboration reported in the experiment of Hawkings and Goldstone.

This paper develops a new approach for estimating the internal model of an autonomous agent that can plan and act, by interrogating it. In this approach, the user may ask an autonomous agent a series of questions, which the agent answers truthfully. Our main contribution is an algorithm that generates an interrogation policy in the form of a sequence of questions to be posed to the agent. Answers to these questions are used to derive a minimal, functionally indistinguishable class of agent models. While asking questions exhaustively for every aspect of the model can be infeasible even for small models, our approach generates questions in a hierarchical fashion to eliminate large classes of models that are inconsistent with the agent. Empirical evaluation of our approach shows that for a class of agents that may use arbitrary black-box transition systems for planning, our approach correctly and efficiently computes STRIPS-like agent models through this interrogation process.

Leonel Garcia-Marques CICPSI Faculdade de Psicologia Universidade de Lisboa Portugal garcia_marques@sapo.pt Abstract --An agent, artificial or human, must be continuously adjusting its behaviour in order to thrive in a more or less demanding environment. An artificial agent with the ability to predict human emotional valence in a geospatial and temporal context requires proper adaptation to its mobile device environment with resource consumption strict restrictions (e.g., power from battery). The developed distributed system includes a mobile device embodied agent ( SensAI) plus Cloud-expanded ( Expanse) cognition and memory resources. The system is designed with several adaptive mechanisms in a best effort for the agent to cope with its interacting humans and to be resilient on collecting data for machine learning towards prediction. These mechanisms encompass homeostatic-like adjustments such as auto recovering from an unexpected failure in the mobile device, forgetting repeated data to save local memory, adjusting actions to a proper moment (e.g., notify only when human is interacting), and the Expanse complementary learning algorithms' parameters with auto adjustments. Regarding emotional valence prediction performance, results from a comparison study between state-of-the-art algorithms revealed Extreme Gradient Boosting on average the best model for prediction with efficient energy use, and explainable using feature importance inspection. Therefore, this work contributes with a smartphone sensing-based system, distributed in the Cloud, robust to unexpected behaviours from humans and the environment, able to predict emotional valence states with very good performance. I NTRODUCTION The scientific evidence of epigenetics reveal on/off mechanisms inside chromosomes of human agents and reinforces the importance of any entity continuous adaptation to its environment.

Deep reinforcement learning (DRL) has made great achievements since proposed. Generally, DRL agents receive high-dimensional inputs at each step, and make actions according to deep-neural-network-based policies. This learning mechanism updates the policy to maximize the return with an end-to-end method. In this paper, we survey the progress of DRL methods, including value-based, policy gradient, and model-based algorithms, and compare their main techniques and properties. Besides, DRL plays an important role in game artificial intelligence (AI). We also take a review of the achievements of DRL in various video games, including classical Arcade games, first-person perspective games and multi-agent real-time strategy games, from 2D to 3D, and from single-agent to multi-agent. A large number of video game AIs with DRL have achieved super-human performance, while there are still some challenges in this domain. Therefore, we also discuss some key points when applying DRL methods to this field, including exploration-exploitation, sample efficiency, generalization and transfer, multi-agent learning, imperfect information, and delayed spare rewards, as well as some research directions.

To make machine exhibit human-like abilities in the domains like robotics and conversation, social commonsense knowledge (SCK), i.e., common sense about social contexts and social roles, is absolutely necessarily. Therefor, our ultimate goal is to acquire large-scale SCK to support much more intelligent applications. Before that, we need to know clearly what is SCK and how to represent it, since automatic information processing requires data and knowledge are organized in structured and semantically related ways. For this reason, in this paper, we identify and formalize three basic types of SCK based on first-order theory. Firstly, we identify and formalize the interrelationships, such as having-role and having-social_relation, among social contexts, roles and players from the perspective of considering both contexts and roles as first-order citizens and not generating role instances. Secondly, we provide a four level structure to identify and formalize the intrinsic information, such as events and desires, of social contexts, roles and players, and illustrate the way of harvesting the intrinsic information of social contexts and roles from the exhibition of players in concrete contexts. And thirdly, enlightened by some observations of actual contexts, we further introduce and formalize the embedding of social contexts, and depict the way of excavating the intrinsic information of social contexts and roles from the embedded smaller and simpler contexts. The results of this paper lay the foundation not only for formalizing much more complex SCK but also for acquiring these three basic types of SCK.

We start with the distinction of outcome- and belief-based Bayesian models of the sequential update of agents' beliefs and subjective reliability of sources (trust). We then focus on discussing the influential Bayesian model of belief-based trust update by Eric Olsson, which models dichotomic events and explicitly represents anti-reliability. After sketching some disastrous recent results for this perhaps most promising model of belief update, we show new simulation results for the temporal dynamics of learning belief with and without trust update and with and without communication. The results seem to shed at least a somewhat more positive light on the communicating-and-trust-updating agents. This may be a light at the end of the tunnel of belief-based models of trust updating, but the interpretation of the clear findings is much less clear.

We present a Spades bidding algorithm that is superior to recreational human players and to publicly available bots. Like in Bridge, the game of Spades is composed of two independent phases, \textit{bidding} and \textit{playing}. This paper focuses on the bidding algorithm, since this phase holds a precise challenge: based on the input, choose the bid that maximizes the agent's winning probability. Our \emph{Bidding-in-Spades} (BIS) algorithm heuristically determines the bidding strategy by comparing the expected utility of each possible bid. A major challenge is how to estimate these expected utilities. To this end, we propose a set of domain-specific heuristics, and then correct them via machine learning using data from real-world players. The \BIS algorithm we present can be attached to any playing algorithm. It beats rule-based bidding bots when all use the same playing component. When combined with a rule-based playing algorithm, it is superior to the average recreational human.

Particle- and agent-based systems are a ubiquitous modeling tool in many disciplines. We consider the fundamental problem of inferring interaction kernels from observations of agent-based dynamical systems given observations of trajectories, in particular for collective dynamical systems exhibiting emergent behaviors with complicated interaction kernels, in a nonparametric fashion, and for kernels which are parametrized by a single unknown parameter. We extend the estimators introduced in \cite{PNASLU}, which are based on suitably regularized least squares estimators, to these larger classes of systems. We provide extensive numerical evidence that the estimators provide faithful approximations to the interaction kernels, and provide accurate predictions for trajectories started at new initial conditions, both throughout the ``training'' time interval in which the observations were made, and often much beyond. We demonstrate these features on prototypical systems displaying collective behaviors, ranging from opinion dynamics, flocking dynamics, self-propelling particle dynamics, synchronized oscillator dynamics, and a gravitational system. Our experiments also suggest that our estimated systems can display the same emergent behaviors of the observed systems, that occur at larger timescales than those used in the training data. Finally, in the case of families of systems governed by a parameterized family of interaction kernels, we introduce novel estimators that estimate the parameterized family of kernels, splitting it into a common interaction kernel and the action of parameters. We demonstrate this in the case of gravity, by learning both the ``common component'' $1/r^2$ and the dependency on mass, without any a priori knowledge of either one, from observations of planetary motions in our solar system.