Reinforcement Learning from Human Feedback (RLHF) has been pivotal in aligning Large Language Models with human values but often suffers from overopti-mization due to its reliance on a proxy reward model.

Despite these strides, evaluating these models poses substantial challenges.

Inverse problems describe the process of estimating the causal factors from a set of measurements or data. Mapping of often incomplete or degraded data to parameters is ill-posed, thus data-driven iterative solutions are required, for example when reconstructing clean images from poor signals.

Our focus in this paper is to solve causal inference problems using only observational data and (potentially partial) knowledge on the causal graph of the underlying structural causal model (SCM).