Although the fairness community has recognized the importance of data, re-searchers inthe area primarily rely on UCIAdult when itcomes totabular data.

Despite these achievements, LLMs often underperform in domain-specific tasks without training on relevant domain data.

In this document we provide the detailed proofs of results presented in the main manuscript. We also provide the Hoeffding decomposition for the bootstrap in Proposition A.4. In Section B we provide all results needed for a complete proof of Theorem 1. In Sections B.1, B.2, and B.3 we provide the proof of Theorem 1, the adaptation of high dimensional CLT of [8] to our setting and all supporting lemmas, respectively. In Section C we provide all details of the proof of the Bootstrap consistency, i.e.

An Zhang and Leheng Sheng contribute equally to this work. Zhibo Cai is the corresponding author.