Rapid Mismatch Estimation via Neural Network Informed Variational Inference