Can an arbitrarily intelligent reinforcement learning agent be kept under control by a human user? Or do agents with sufficient intelligence inevitably find ways to shortcut their reward signal? This question impacts how far reinforcement learning can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we use an intuitive yet precise graphical model called causal influence diagrams to formalize reward tampering problems. We also describe a number of modifications to the reinforcement learning objective that prevent incentives for reward tampering. We verify the solutions using recently developed graphical criteria for inferring agent incentives from causal influence diagrams. Along the way, we also compare corrigibility and self-preservation properties of the various solutions, and discuss how they can be combined into a single agent without reward tampering incentives.

The Expectation-Maximization (EM) algorithm is a fundamental tool in unsupervised machine learning. It is often used as an efficient way to solve Maximum Likelihood (ML) estimation problems, especially for models with latent variables. It is also the algorithm of choice to fit mixture models: generative models that represent unlabelled points originating from $k$ different processes, as samples from $k$ multivariate distributions. In this work we define and use a quantum version of EM to fit a Gaussian Mixture Model. Given quantum access to a dataset of $n$ vectors of dimension $d$, our algorithm has convergence and precision guarantees similar to the classical algorithm, but the runtime is only polylogarithmic in the number of elements in the training set, and is polynomial in other parameters - as the dimension of the feature space, and the number of components in the mixture. We generalize further the algorithm in two directions. First, we show how to fit any mixture model of probability distributions in the exponential family. Then, we show how to use this algorithm to compute the Maximum a Posteriori (MAP) estimate of a mixture model: the Bayesian approach to likelihood estimation problems. We discuss the performance of the algorithm on datasets that are expected to be classified successfully by those algorithms, arguing that on those cases we can give strong guarantees on the runtime.

In many application settings involving networks, such as messages between users of an on-line social network or transactions between traders in financial markets, the observed data are in the form of relational events with timestamps, which form a continuous-time network. We propose the Community Hawkes Independent Pairs (CHIP) model for community detection on such timestamped relational event data. We demonstrate that applying spectral clustering to adjacency matrices constructed from relational events generated by the CHIP model provides consistent community detection for a growing number of nodes. In particular, we obtain explicit non-asymptotic upper bounds on the misclustering rates based on the separation conditions required on the parameters of the model for consistent community detection. We also develop consistent and computationally efficient estimators for the parameters of the model. We demonstrate that our proposed CHIP model and estimation procedure scales to large networks with tens of thousands of nodes and provides superior fits compared to existing continuous-time network models on several real networks.

In science and engineering, it is an ubiquitous problem to estimate the probability of event based on Monte Carlo simulation. For instance, in engineering technology, a critical c oncern is the probability of failure or risk, which is generally considered as the probability that certain pre -specified requirements for the relevant system are violated in the presence of uncertainties. Ever since th e advent of modern computers, extensive research works have been devoted to quantitative approaches o f risk evaluation for engineering systems (see, e.g., [1, 8, 9, 11, 16, 18, 20] and the references therein). I n additional to theoretical development, many softwares have been developed for risk evaluation. For exam ple, for control systems, a software called RACT has been developed for evaluating the risk of uncertain syste ms [7, 21]. Many softwares such as APMC [13], PRISM [15], UPPAAL [6], have been developed for evaluating t he risk of stochastic discrete event systems (see, [1] and the references therein). One of the remarkable achievements of existing theories and softw ares is the rigorous control of error in the estimation of probability, that is, the probability of relevant ev ent can be evaluated with certified reliability. Theoretically, for a priori given α, δ (0, 1), existing methods are able to produce an estimate null p for the true value of the probability p so that one can be 100(1 δ)% confident that null p p α holds. 1 Unfortunately, existing methods suffer from huge computational complexity as the margin of absolute error α is small, e.g. 10

Learning with label proportions(LLP), which seeks an instance-level classifier merely based on bag-level label proportions, is a new paradigm in machine learning that addresses the classification of instances [1, 2, 3]. In LLP, we only know the proportions of examples belonging to different classes in each bag; however the labels of the instances are unknown. From the binary classification perspective, the task of LLP is to learn a classifier to classify the unknown label instance as either positive class or negative class. The formulation that learning with label proportions has been first proposed by Kuck et al. in [1], which can be used for political elections analysis. In the case of politician polls, each candidate may have a group of loyal voters and some swing voters. They may know the vague proportion of votes cast in each district; however, they usually do not know the vote of each person. Since the candidates have limited resources, they have to analyze political elections and consider which kind of voters they should focus on so as to maximize their interests. To date, LLP has been applied to forecasting revenue [4], image classification [5, 6], video event detection [7], demographics mining [8] and privacy protection [9]. Figure 1 illustrates the binary classification problem in LLP.

Question answering over knowledge graphs (KGQA) has evolved from simple single-fact questions to complex questions that require graph traversal and aggregation. We propose a novel approach for complex KGQA that uses unsupervised message passing, which propagates confidence scores obtained by parsing an input question and matching terms in the knowledge graph to a set of possible answers. First, we identify entity, relationship, and class names mentioned in a natural language question, and map these to their counterparts in the graph. Then, the confidence scores of these mappings propagate through the graph structure to locate the answer entities. Finally, these are aggregated depending on the identified question type. This approach can be efficiently implemented as a series of sparse matrix multiplications mimicking joins over small local subgraphs. Our evaluation results show that the proposed approach outperforms the state-of-the-art on the LC-QuAD benchmark. Moreover, we show that the performance of the approach depends only on the quality of the question interpretation results, i.e., given a correct relevance score distribution, our approach always produces a correct answer ranking. Our error analysis reveals correct answers missing from the benchmark dataset and inconsistencies in the DBpedia knowledge graph. Finally, we provide a comprehensive evaluation of the proposed approach accompanied with an ablation study and an error analysis, which showcase the pitfalls for each of the question answering components in more detail.

Test automation can result in reduction in cost and human effort. If the optimal policy, the course of actio ns taken, for the intended objective in a testing process could be learnt by the testing system (e.g., a smart tester agent), then it could be reused in similar situations, thus leading to higher efficiency, i.e., less computational time. Automating stress testing to find performance breaking points remains a challenge for complex software systems. Common approaches are mainly based on source code or system model analysis or use - case based techniques. However, source code or system models might not be avai lable at testing time. In this paper, we propose a self - adaptive fuzzy reinforcement learning - based performance (stress) testing framework (SaFReL) that enables the tester agent to learn the optimal policy for generating stress test case s leading to performance breaking point without access to performance model of the system under test. SaFReL learns the optimal policy through an initial learning, then reuses it during a transfer learning phase, while keeping the learning running in the long - term. Through multiple experiments on a simulated environment, we demonstrate that our approach generates the stress test case s for different programs efficiently and adaptively without access to performance models .

When several predictors have been trained to solve the same classification task, a second level of algorithmic procedure is necessary to reconcile the classifier predictions and deliver a single one. Such a procedure is known as classifier combination, fusion or aggregation. When each individual classifier is trained using the same training algorithm (but under different circumstances) the aggregation procedure is referred to as an ensemble method. When each classifier may be generated by different training algorithms, the aggregation procedure is referred to as a multiple classifier system. In both cases, the set of individual classifiers is called a classifier ensemble. Classifier combination comes either from a choice of the programmer or is imposed by context. In the first case, combination is meant to increase classification performances by either increasing the learning capacity or mitigating 1 arXiv:1908.06475v1

YY, ZZZZ 1 Music Transcription Based on Bayesian Piece-Specific Score Models Capturing Repetitions Eita Nakamura, Kazuyoshi Y oshii, Member, IEEE Abstract --Most work on models for music transcription has focused on describing local sequential dependence of notes in musical scores and failed to capture their global repetitive structure, which can be a useful guide for transcribing music. Focusing on the rhythm, we formulate several classes of Bayesian Markov models of musical scores that describe repetitions indirectly by sparse transition probabilities of notes or note patterns. This enables us to construct piece-specific models for unseen scores with unfixed repetitive structure and to derive tractable inference algorithms. Moreover, to describe approximate repetitions, we explicitly incorporate a process of modifying the repeated notes/note patterns. We apply these models as a prior music language model for rhythm transcription, where piece-specific score models are inferred from performed MIDI data by unsupervised learning, in contrast to the conventional supervised construction of score models. Evaluations using vocal melodies of popular music showed that the Bayesian models improved the transcription accuracy for most of the tested model types, indicating the universal efficacy of the proposed approach. I NTRODUCTION Music transcription is an actively studied but yet unsolved problem in music information processing [1], [2]. One of the goals of music transcription is to convert a music performance signal into a human-readable symbolic musical score. While recent studies have achieved highly accurate pitch detection [3]-[7], it is also necessary to transcribe rhythms in order to obtain symbolic music representation [8]-[18]. Since there are many logically possible representations of rhythms (including meaningless one for humans) for a given performance [11], using a score model that describes prior knowledge about musical scores is a key to solve this problem. A common approach for music transcription is to integrate a musical score (language) model and a performance/acoustic model to obtain a proper transcription that best fits an input performance signal, similarly to the method of statistical speech recognition. More recently, end-to-end approaches have also been attempted [19]-[21], which have been of limited success so far. Manuscript received XX, YY; revised XX, YY . This work was supported partially by JSPS KAKENHI (Nos. The work of EN was supported by the JSPS research fellowship (PD).

Although we have focused on the "hot" area of A Vs, our discussion and the novel CBI theorems are more generally applicable. We see them as especially useful now for MLbased systems with critical applications, although not with extreme requirements, since assurance in these systems must rely on combinations of statistical evidence with other verification methods that are, as yet, not well-established. A PPENDIX A. Statement And Proof of CBI Theorem 1 Problem: Consider the set D of all probability distributions defined over the unit interval, each distribution representing a potential prior distribution of pfm values for an A V . For 0 p l null null 1, we seek a prior distribution that minimises the posterior confidence in a reliability bound p [ p l, 1], given k fatalities have occurred over n miles driven and subject to constraints on some quantiles of the prior distribution. That is, for θ (0, 1], we solve minimise D Pr ( X null p k & n) subject to Pr ( X null null) θ, Pr (X null p l) 1 Solution: There is a prior in D that minimises the posterior confidence: the 2-point distribution Pr ( X x) θ 1 x x 1 (1 θ)1 x x 3 where p l null x 1 null null x 3, and the values of x 1 and x 3 both depend on the model parameters (i.e.