SO(3)-invariant PCA with application to molecular data