It is common that these data themselves have inherently complicated 3D geometric structures.

It is common that these data themselves have inherently complicated 3D geometric structures.

The objective is to predict potential energy and forces.

Nevertheless, it remains elusive that whether the Transformer-based architecture can do molecular modeling as good as equivariant GNNs.

In general, this has a cubic cost in dataset size and is sensitive to conditioning.

The contributions of this paper are as follows (Figure 1): 1.

The proposed implementation of Gunsilius' algorithm computes For example, in the expenditure dataset (see Section I.3), In Figure 4, we show the results of Gunsilius's algorithm for three different Note that this algorithm works on the empirical CDFs of all variables, i.e., they are all scaled to lie Figure 4: We show results of Gunsilius's algorithm for 3 different settings of The practical issue of course is the optimization. That alone is already very computationally demanding and has convergence problems. A practical resource, sample size, limits the representational size of the estimator. How to achieve "enough variability" without aiming at a completely flexible distribution of In any case, the finite mixture of Gaussians approach can still be implemented with the reparameter-ization trick. The relation to Gunsilius algorithm is that our "base measure" is smoothly adaptive, leading to possibly more stable behavior in practice.

In-context learning (ICL) refers to a remarkable capability of pretrained large language models, which can learn a new task given a few examples during inference. However, theoretical understanding of ICL is largely under-explored, particularly whether transformers can be trained to generalize to unseen examples in a prompt, which will require the model to acquire contextual knowledge of the prompt for generalization. This paper investigates the training dynamics of transformers by gradient descent through the lens of non-linear regression tasks. The contextual generalization here can be attained via learning the template function for each task in-context, where all template functions lie in a linear space with $m$ basis functions. We analyze the training dynamics of one-layer multi-head transformers to {in-contextly} predict unlabeled inputs given partially labeled prompts, where the labels contain Gaussian noise and the number of examples in each prompt are not sufficient to determine the template. Under mild assumptions, we show that the training loss for a one-layer multi-head transformer converges linearly to a global minimum. Moreover, the transformer effectively learns to perform ridge regression over the basis functions. To our knowledge, this study is the first provable demonstration that transformers can learn contextual (i.e., template) information to generalize to both unseen examples and tasks when prompts contain only a small number of query-answer pairs.