In this paper, we explore methods for comparing agent behavior with human behavior to assist with validation. Our exploration begins by considering a simple method of behavior comparison. Motivated by shortcomings in this initial approach, we introduce behavior bounding, an automated model-based approach for comparing behavior that is inspired, in part, by Mitchells Version Spaces. We show that behavior bounding can be used to compactly represent both human and agent behavior. We argue that relatively low amounts of human effort are required to build, maintain, and use the data structures that underlie behavior bounding, and we provide a theoretical basis for these arguments using notions of PAC Learnability. Next, we show empirical results indicating that this approach is effective at identifying differences in certain types of behaviors and that it performs well when compared against our initial benchmark methods. Finally, we demonstrate that behavior bounding can produce information that allows developers to identify and fix problems in an agents behavior much more efficiently than standard debugging techniques.
Statistical inference using pairwise comparison data has been an effective approach to analyzing complex and sparse networks. In this paper we propose a general framework for modeling the mutual interaction in a probabilistic network, which enjoys ample flexibility in terms of parametrization. Within this set-up, we establish that the maximum likelihood estimator (MLE) for the latent scores of the subjects is uniformly consistent under a near-minimal condition on network sparsity. This condition is sharp in terms of the leading order asymptotics describing the sparsity. The proof utilizes a novel chaining technique based on the error-induced metric as well as careful counting of comparison graph structures. Our results guarantee that the MLE is a valid estimator for inference in large-scale comparison networks where data is asymptotically deficient. Numerical simulations are provided to complement the theoretical analysis.
We consider sequential or active ranking of a set of n items based on noisy pairwise comparisons. Items are ranked according to the probability that a given item beats a randomly chosen item, and ranking refers to partitioning the items into sets of pre-specified sizes according to their scores. This notion of ranking includes as special cases the identification of the top-k items and the total ordering of the items. We first analyze a sequential ranking algorithm that counts the number of comparisons won, and uses these counts to decide whether to stop, or to compare another pair of items, chosen based on confidence intervals specified by the data collected up to that point. We prove that this algorithm succeeds in recovering the ranking using a number of comparisons that is optimal up to logarithmic factors. This guarantee does not require any structural properties of the underlying pairwise probability matrix, unlike a significant body of past work on pairwise ranking based on parametric models such as the Thurstone or Bradley-Terry-Luce models. It has been a long-standing open question as to whether or not imposing these parametric assumptions allows for improved ranking algorithms. For stochastic comparison models, in which the pairwise probabilities are bounded away from zero, our second contribution is to resolve this issue by proving a lower bound for parametric models. This shows, perhaps surprisingly, that these popular parametric modeling choices offer at most logarithmic gains for stochastic comparisons.
Grigor Dimitrov is "tired" of being compared to Roger Federer and called it is a thing of the past. The Bulgarian is keen to move past the Swiss ace's shadow as he looks to finally fulfill his potential that led to him being called "Baby Fed." The 26-year-old was compared to the arguably the greatest player of all time mainly owing to their playing style – the fluency in the strokes and the single handed backhand, but in terms of success it can be said he is yet to show his true potential. Dimitrov had a phenomenal 2017 season after two seasons without a single title. He won four titles last campaign which included the ATP Finals in London, that saw him achieve his best ever ranking – world number three – and was just behind Rafael Nadal and Federer, who had a combined 13 titles between them including equally sharing the four Grand Slams.
MPs on the Energy Committee have written to the new business secretary to demand no change to the current rules on price comparison websites. As part of its recent report on the energy market, the Competition and Markets Authority (CMA) said that such websites would no longer be required to show all the deals on offer. But the MPs said that would undermine consumer trust, and harm competition. The CMA rules will allow sites only to show deals on which they earn money. This reverses a decision by the regulator Ofcom.