Energy-Based Processes for Exchangeable Data

Many machine learning problems consider data where each instance is, itself, an unordered set of elements; i.e., such that each observation is a set. Data of this kind arises in a variety of applications, ranging from document modeling (Blei et al., 2003; Garnelo et al., 2018a) and multi-task learning (Zaheer et al., 2017; Edwards & Storkey, 2016; Liu et al., 2019) to 3D point cloud modeling (Li et al., 2018; Yang et al., 2019). In unsupervised settings, a dataset typically consists of a set of such sets, while in supervised learning, it consists of a set of (set, label) pairs. Modeling a distribution over a space of instances, where each instance is, itself, an unordered set of elements involves two key considerations: (1) the elements within a single instance are exchangeable, i.e., the elements are order invariant; and (2) the cardinalities of the instances (sets) vary, i.e., instances need not exhibit the same cardinality. Modeling both unconditional and conditional distributions over instances (sets) are relevant to consider, since these support unsupervised and supervised tasks respectively. For unconditional distribution modeling, there has been significant prior work on modeling set distributions, which has sought to balance competing needs to expand model flexibility and preserve tractability on the one hand, with respecting exchangeability and varying instance cardinalities on the other hand. However, managing these tradeoffs has proved to be quite difficult, and current approaches remain limited in different respects. For example, a particularly straightforward strategy for modeling distributions over instances x {x 1,..., x n } without assuming fixed cardinality is simply to use a recurrent neural network (RNNs) to encode instance probability auto-regressively via p (x) n