We have recently seen great progress in learning interpretable music representations, ranging from basic factors, such as pitch and timbre, to high-level concepts, such as chord and texture. However, most methods rely heavily on music domain knowledge. It remains an open question what general computational principles interpretable representations, especially low-dim factors that agree with human perception. In this study, we take inspiration from modern physics and use as a self-consistency constraint for the latent space. Specifically, it requires the prior model that characterises the dynamics of the latent states to be with respect to certain group transformations. We show that physical symmetry leads the model to learn a pitch factor from unlabelled monophonic music audio in a self-supervised fashion. In addition, the same methodology can be applied to computer vision, learning a 3D Cartesian space from videos of a simple moving object without labels. Furthermore, physical symmetry naturally leads to, a new technique which improves sample efficiency.