Neural Information Processing Systems http://nips.cc/

The impossibility theorem of fairness is a foundational result in the algorithmic fairness literature. It states that outside of special cases, one cannot exactly and simultaneously satisfy all three common and intuitive definitions of fairness demographic parity, equalized odds, and predictive rate parity. This result has driven most works to focus on solutions for one or two of the metrics.

For more reliable comparison, we repeat experiments for100random seedsinstead of 10. "init tune" denotes tuningσ and choosing betweenN or U (see Figure 1 at the bottom); tuning isdone in the same wayasforotherhyperparameters. We will also add results of GCN supporting our conclusions (Table 115 and Figure 1). Note20 that in Table 1 of the submitted paper, forCOLORSand MNIST-75sp,21 ChebyGINs are equivalent to ChebyNets as described in Table 1 of22 theSupplementary material and elaborated onfollowing that table (see23 footnote3). In our model, the features are25 weighted by attention scores according to Eq. 3, so it is soft. In this26 case, the features indeed reduce their scale.

Minimizing the inference cost and latency of foundation models has become a crucial area of research. Optimization approaches include theoretically lossless methods and others without accuracy guarantees like quantization. In all of these cases it is crucial to ensure that the model quality has not degraded. However, even at temperature zero, model generations are not necessarily robust even to theoretically lossless model optimizations due to numerical errors. We thus require statistical tools to decide whether a finite-sample accuracy deviation is an evidence of a model's degradation or whether it can be attributed to (harmless) noise in the evaluation. We propose a statistically sound hypothesis testing framework based on McNemar's test allowing to efficiently detect model degradations, while guaranteeing a controlled rate of false positives. The crucial insight is that we have to confront the model scores on each sample, rather than aggregated on the task level. Furthermore, we propose three approaches to aggregate accuracy estimates across multiple benchmarks into a single decision. We provide an implementation on top of the largely adopted open source LM Evaluation Harness and provide a case study illustrating that the method correctly flags degraded models, while not flagging model optimizations that are provably lossless. We find that with our tests even empirical accuracy degradations of 0.3% can be confidently attributed to actual degradations rather than noise.