Semi-parametric $\gamma$-ray modeling with Gaussian processes and variational inference