A large part of the current success of deep learning lies in the effectiveness of data - more precisely: labelled data. Yet, labelling a dataset with human annotation continues to carry high costs, especially for videos. While in the image domain, recent methods have allowed to generate meaningful (pseudo-) labels for unlabelled datasets without supervision, this development is missing for the video domain where learning feature representations is the current focus.

A common approach [15, 43, 61] is to learn a shared feature extractor, along with domain-specific classifier modules (Figure 1a), which yield an ensemble prediction for the target samples.

First, let us recall that AIS is based on a simulated annealing process where a configuration is gradually brought from temperature T = to T = 1 using a set of bridging distributions. Foreach temperature, we define the transition operator, Tk(v0,v) to bring a configuration v to v0 varying the temperature according to the temperature schedule. In our case it is done using MC sampling layer-wise. In our work, we used a set of Nβ [104,105] temperatures uniformly distributed in this interval (dependingonthesystemsize). Inpractice,oneobservesthatERBM goesbelowED atlong sampling times if the machine was trained out of equilibrium.

Inparticular,weshowthat using the popular k (persistent) contrastive divergence approaches, with k small, the dynamics of the learned model are extremely slow and often dominated by strong out-of-equilibrium effects.