ELF: Exact-Lipschitz Based Universal Density Approximator Flow

Normalizing flows have become more popular within the last few years; however, they continue to have limitations compared to other generative models, more specifically that they are computationally expensive in terms of memory and time. Early implementations of Normalizing Flows were coupling layers (Dinh et al., 2014, 2017; Kingma and Dhariwal, 2018) and autoregressive flows (Papamakarios et al., 2017; Kingma et al., 2016). These have easy to compute log-likelihoods; however, coupling layers tend to need quite a few parameters to achieve strong performance and autoregressive flows are extremely expensive to sample from. The newer technique of residual flows (Chen et al., 2019) allows for models that are built on standard components and have inductive biases that favor simpler functions (Gopal, 2020); however, these have the problem of being expensive in terms of time for computing log-likelihoods and training, as well as require quite a few layers for strong performance. Since the introduction of these models, there have been many developments that have lead to improvement in parameter efficiency such as FFJORD (Grathwohl et al., 2019), a continuous normalizing flow, that has a dynamic number of layers. However, this too can have computational problems as having a few dynamics layers can lead to hundreds of implicit layers. Among the flows introduced, the ones with provable universal approximation capability are Affine Coupling Layers (Dinh et al., 2014, 2017; Teshima et al., 2020), Neural Autoregressive Flows (NAF, Huang et al. (2018)), Block NAFs (BNAF, Cao et al. (2019)), Sum-of-Squares Polynomial Flow (Jaini et al., 2019), and Convex Potential Flows (CP-Flow, Huang et al. (2021)). Though these have been shown to be universal approximators, they do not necessarily translate into faster, more efficient training, and some of the flows listed require the expensive sampling routine of autoregressive flows.