When dealing with complex data, the effectiveness of a classifier/predictor is limited by its ability to extract useful information. As such, representations that clearly expose the semantics of the data should then be most amenable to downstream learning [1, 2]. This is often referred to as a challenge of acquiring a disentangled representation over the factors of the data [3]. A popular recent trend that has had significant success in this regard uses semi-supervised Variational AutoEncoders (VAE) [4, 5, 6, 7, 8, 9]. Whilst fully unsupervised VAE methods have been shown to require strong inductive bias [10], semi-supervised methods achieve disentanglement by training additional auxiliary tasks that are defined on the factors, alongside the standard VAE objective (see Appendix Eqn. 3).