In this paper, we consider the problem of fair statistical inference involving outcome variables. Examples include classification and regression problems, and estimating treatment effects in randomized trials or observational data. The issue of fairness arises in such problems where some covariates or treatments are "sensitive," in the sense of having potential of creating discrimination. In this paper, we argue that the presence of discrimination can be formalized in a sensible way as the presence of an effect of a sensitive covariate on the outcome along certain causal pathways, a view which generalizes (Pearl, 2009). A fair outcome model can then be learned by solving a constrained optimization problem. We discuss a number of complications that arise in classical statistical inference due to this view and provide workarounds based on recent work in causal and semi-parametric inference.

Human susceptibility to epidemics of misinformation and disease has grown manyfold as the world we inhabit keeps getting smaller due to increased access to online information and soaring global mobility. Social media platforms have changed the way we consume information [Schmidt et al., 2017], and more and more people find their news through social media [Newman et al., 2015]. Following the 2016 presidential election in the United States, there have been investigations into the spread of false stories, or "fake news" on social media, and based on web browsing data, archives of fact-checking websites, and results from an online survey, a recent study found that social media were an important source of election news [Allcott and Gentzkow, 2017]. While ascertainment of social network structures is generally difficult using traditional survey-based approaches, such as name generators, which are survey questions designed to solicit information about friends and acquaintances of a subject, online platforms readily capture the structure of large-scale social networks, therefore making them well suited to study spread of information whether accurate or not. Further, although the transmission mechanisms are very different, the spread of information in online systems has many similarities to the spread of infectious diseases among hosts in a population. From a mathematical and statistical point of view, one can therefore investigate the spread of pathogens and the spread of information in the same framework as long as the network structure accurately captures the transmission pathways and the spreading process is parametrized appropriately. In this paper, we consider a simple susceptible-infected (SI) process and a more complex spreading process on a fixed and known network structure. This spreading process may be conceptualized as propagating either a pathogen or a piece of information. We focus on addressing two distinct questions that are relevant in both settings: (1) How to infer the unknown parameters associated with the spreading process?

We suggest a general approach to quantification of different forms of uncertainty in regression tasks performed by artificial neural networks. It is based on the simultaneous training of two neural networks with a joint loss function. One of the networks performs predictions and the other simultaneously quantifies the uncertainty of predictions by estimating the locally averaged loss of the first one. Unlike in many classical uncertainty quantification methods, the targets are not assumed to be sampled from a probability distribution of an a priori given form. We analyze how the hyperparameters affect the learning process and, additionally, show that our method even allows for better predictions compared to standard neural networks without uncertainty counterparts. Finally, we show that particular cases of our approach include maximization of log-likelihood, assuming Gaussian or Laplace noise.

We present a mixed multinomial logit (MNL) model, which leverages the truncated stickbreaking process representation of the Dirichlet process as a flexible nonparametric mixing distribution. The proposed model is a Dirichlet process mixture model and accommodates discrete representations of heterogeneity, like a latent class MNL model. Yet, unlike a latent class MNL model, the proposed discrete choice model does not require the analyst to fix the number of mixture components prior to estimation, as the complexity of the discrete mixing distribution is inferred from the evidence. For posterior inference in the proposed Dirichlet process mixture model of discrete choice, we derive an expectation maximisation algorithm. In a simulation study, we demonstrate that the proposed model framework can flexibly capture differently-shaped taste parameter distributions. Furthermore, we empirically validate the model framework in a case study on motorists' route choice preferences and find that the proposed Dirichlet process mixture model of discrete choice outperforms a latent class MNL model and mixed MNL models with common parametric mixing distributions in terms of both in-sample fit and out-of-sample predictive ability. Compared to extant modelling approaches, the proposed discrete choice model substantially abbreviates specification searches, as it relies on less restrictive parametric assumptions and does not require the analyst to specify the complexity of the discrete mixing distribution prior to estimation. 2 1. Introduction

Learning the structure of graphical models from data usually incurs a heavy curse of dimensionality that renders this problem intractable in many real-world situations. The rare cases where the curse becomes a blessing provide insight into the limits of the efficiently computable and augment the scarce options for treating very under-sampled, high-dimensional data. We study a special class of Gaussian latent factor models where each (non-iid) observed variable depends on at most one of a set of latent variables. We derive information-theoretic lower bounds on the sample complexity for structure recovery that suggest complexity actually decreases as the dimensionality increases. Contrary to this prediction, we observe that existing structure recovery methods deteriorate with increasing dimension. Therefore, we design a new approach to learning Gaussian latent factor models that benefits from dimensionality. Our approach relies on an unconstrained information-theoretic objective whose global optima correspond to structured latent factor generative models. In addition to improved structure recovery, we also show that we are able to outperform state-of-the-art approaches for covariance estimation on both synthetic and real data in the very under-sampled, high-dimensional regime.

Stochastic Gradient Descent with a constant learning rate (constant SGD) simulates a Markov chain with a stationary distribution. With this perspective, we derive several new results. (1) We show that constant SGD can be used as an approximate Bayesian posterior inference algorithm. Specifically, we show how to adjust the tuning parameters of constant SGD to best match the stationary distribution to a posterior, minimizing the Kullback-Leibler divergence between these two distributions. (2) We demonstrate that constant SGD gives rise to a new variational EM algorithm that optimizes hyperparameters in complex probabilistic models. (3) We also propose SGD with momentum for sampling and show how to adjust the damping coefficient accordingly. (4) We analyze MCMC algorithms. For Langevin Dynamics and Stochastic Gradient Fisher Scoring, we quantify the approximation errors due to finite learning rates. Finally (5), we use the stochastic process perspective to give a short proof of why Polyak averaging is optimal. Based on this idea, we propose a scalable approximate MCMC algorithm, the Averaged Stochastic Gradient Sampler.

The motivations for using variational inference (VI) in neural networks differ significantly from those in latent variable models. This has a counter-intuitive consequence; more expressive variational approximations can provide significantly worse predictions as compared to those with less expressive families. In this work we make two contributions. First, we identify a cause of this performance gap, variational over-pruning. Second, we introduce a theoretically grounded explanation for this phenomenon. Our perspective sheds light on several related published results and provides intuition into the design of effective variational approximations of neural networks.

This chapter provides a self-contained introduction to the use of Bayesian inference to extract large-scale modular structures from network data, based on the stochastic block model (SBM), as well as its degree-corrected and overlapping generalizations. We focus on nonparametric formulations that allow their inference in a manner that prevents overfitting, and enables model selection. We discuss aspects of the choice of priors, in particular how to avoid underfitting via increased Bayesian hierarchies, and we contrast the task of sampling network partitions from the posterior distribution with finding the single point estimate that maximizes it, while describing efficient algorithms to perform either one. We also show how inferring the SBM can be used to predict missing and spurious links, and shed light on the fundamental limitations of the detectability of modular structures in networks.

A principled approach to characterize the hidden structure of networks is to formulate generative models, and then infer their parameters from data. When the desired structure is composed of modules or "communities", a suitable choice for this task is the stochastic block model (SBM), where nodes are divided into groups, and the placement of edges is conditioned on the group memberships. Here, we present a nonparametric Bayesian method to infer the modular structure of empirical networks, including the number of modules and their hierarchical organization. We focus on a microcanonical variant of the SBM, where the structure is imposed via hard constraints, i.e. the generated networks are not allowed to violate the patterns imposed by the model. We show how this simple model variation allows simultaneously for two important improvements over more traditional inference approaches: 1. Deeper Bayesian hierarchies, with noninformative priors replaced by sequences of priors and hyperpriors, that not only remove limitations that seriously degrade the inference on large networks, but also reveal structures at multiple scales; 2. A very efficient inference algorithm that scales well not only for networks with a large number of nodes and edges, but also with an unlimited number of modules. We show also how this approach can be used to sample modular hierarchies from the posterior distribution, as well as to perform model selection. We discuss and analyze the differences between sampling from the posterior and simply finding the single parameter estimate that maximizes it. Furthermore, we expose a direct equivalence between our microcanonical approach and alternative derivations based on the canonical SBM.

Many network systems lack a natural low-dimensional embedding from which we can readily extract their most prominent large-scale features. Instead, we have to infer this information from data, typically by decomposing the observed network into modules [1]. A principled approach to perform this task is to formulate generative models that allow this modular decomposition to be found via statistical inference [2]. The most fundamental model used for this purpose is the stochastic block model (SBM) [3], which groups nodes according to their probabilities of connection to the rest of the network. However, a central limitation of most SBM implementations is that they are defined strictly for simple or multigraphs. This means that they do not incorporate extra information on the edges, which are typically present in a variety of systems, and are required for an accurate representation of their structure. For example, to the existence of a route between two airports is associated a distance, to the biomass flow between two species in a food web is associated a flow magnitude, etc. In this work, we develop variations of the SBM that allow for this type of information on the edges to be incorporated into the network model and guide the partition of the nodes into groups in a statistically meaningful way. We follow the same basic idea put forth by Aicher et al. [4], who adapted the SBM to weighted networks by including edge values as additional covariates.