Conditional flow matching for physics-constrained inverse problems with finite training data