This framework has the flexibility of deriving distribution-and algorithm-dependent bounds, which are often tighter than VC-related uniform convergence bounds.

We experimented on using Sigmoid functions, and it does not work. To Reviewer #3: Our algorithm scales linearly w.r.t. the input size, which is not larger than any other algorithms.12 Thiscanberealized42 by a max operation, which is differentiable. Note that we don't need to use argmax operation to find the indexr.43 Instead, every decoded token is involved in the later computation.

However, this lower bound only holds against deterministic preconditioners, and in many contexts randomization is crucial to the success of preconditioners. We prove a stronger lower bound that rules out randomized preconditioners.