The task is to i) predict the unknown parameters, then ii) solve the optimization problem using the predicted parameters, such that the resulting solutions are good even under true parameters.

BLUE (Biomedical Language Understanding Evaluation) is abenchmarkfor 10 datasetsrepresenting 5 tasks [34]. BLURB (Biomedical Language Understanding and Reasoning Benchmark) includes 13 datasetsand 7 tasks [19].

WefirstclusterthequeriesQusingtheK-means clustering to outputS which indicates the membership of queries to different clusters. The lower half of the figure shows the new valueˆVt computed by sparse dot-products with the keysK and values V corresponding tothe the top-k keys inT. Figure 6: We show training/validation loss convergence for different transformer variants. Both the clustered variants are have a significantly better convergence than bothlsh-1 and lsh-4. Note that due to a smaller batch sizefullmakesmanymoreupdates than allother transformer variants. In figure 6a, we show the training loss convergence for different transformer variants.

Clustered attention makes use of similarities between queries and groups them in order to reduce the computational cost. In particular, we perform fast clustering using locality-sensitive hashing and K-Means and only compute the attention once per cluster.