Semi-supervised Invertible Neural Operators for Bayesian Inverse Problems