Sarah Rastegar, Hazel Doughty, Cees G. M. Snoek University of Amsterdam

Some definitions might overlap with the notations in the main paper. We denote the input random variable with X and the category random variable with C. The category code random variable, which we define as the embedding sequence of input X The length of each sequence z, which we show with l(z), equals the number of digits present in that sequence. It measures the randomness of values of X when we only have knowledge about its distribution P. It also measures the minimum number of bits required on average to transmit or encode the values drawn from this probability distribution [1, 2]. The conditional entropy of a random variable X given random variable Z is shown by H(X|Z), which states the amount of randomness we expect to see from X after observing Z. Note that contrary to H(X|Z), mutual information is symmetric. Similar to Shannon's information theory, Kolmogorov Complexity or Algorithmic Information Theory[3-5] measures the shortest length to describe an object. Their difference is that Shannon's information considers that the objects can be described by the characteristic of the source that produces them, but Kolmogorov Complexity considers that the description of each object in isolation can be used to describe it with minimum length.