Pre-trained Graphformer-based Ranking at Web-scale Search (Extended Abstract)

Although graphformer[Yang et al., 2021] has been proposed to combine advantages from GNNs and Both Transformer and Graph Neural Networks Transformers for representation learning with textual graphs, (GNNs) have been employed in the domain of learning there still lack of joint efforts from the two domains (i.e., to rank (LTR). However, these approaches adhere query-webpage pairs and graphs) in LTR. In order to improve to two distinct yet complementary problem the performance of over-parameterized models like formulations: ranking score regression based on Transformers or GNNs, the paradigm of pre-training and query-webpage pairs, and link prediction within fine-tuning has been extensively employed[Liao et al., 2024; query-webpage bipartite graphs, respectively. While Chen et al., 2024g; Chen et al., 2022; Song et al., 2024; it is possible to pre-train GNNs or Transformers on Lyu et al., 2023]. This involves firstly training the models source datasets and subsequently fine-tune them on on large-scale source datasets in an unsupervised or selfsupervised sparsely annotated LTR datasets, the distributional manner to develop their core representation learning shifts between the pair-based and bipartite graph capabilities [Qiang et al., 2023; Xiong et al., 2024a; domains present significant challenges in integrating Xiong et al., 2024b; Lyu et al., 2020]. Subsequently, the pretrained these heterogeneous models into a unified LTR models can be fine-tuned using a small number of annotated framework at web scale. To address this, we introduce samples from the target datasets [Kirichenko et al., 2022; the novel MPGraf model, which leverages Huang et al., 2021; Chen et al., 2023e; Chen et al., 2023d; a modular and capsule-based pre-training strategy, Chen et al., 2023b]. However, such paradigm could not be aiming to cohesively integrate the regression capabilities easily followed by the LTR models leveraging both querywebpage of Transformers with the link prediction pairs and graphs together.