r/MachineLearning - [D] Machine Learning - WAYR (What Are You Reading) - Week 71

MINE: Mutual Information Neural Estimation: The idea is quite simple - take a function that serves as a lower (or upper) bound for a quantity you want to estimate and use gradient ascent (or descent) to maximize (or minimize) the function. In this case, the function is a lower bound on KL divergence: Donsker-Varadhan representation. I have been thinking that this could become a more generalized approach to estimate values that are hard to evaluate from data. Basically take any inequality that involves the term we want to estimate, parameterize it and apply gradient descent/ascent.