Transformers are Efficient In-Context Estimators for Wireless Communication

Department of Electrical and Computer Engineering, Texas A&M University Abstract Pre-trained transformers can perform in-context learning, where they adapt to a new task using only a small number of prompts without any explicit model optimization. Inspired by this attribute, we propose a novel approach, called in-context estimation, for the canonical communication problem of estimating transmitted symbols from received symbols. A communication channel is essentially a noisy function that maps transmitted symbols to received symbols, and this function can be represented by an unknown parameter whose statistics depend on an (also unknown) latent context. Conventional approaches typically do not fully exploit hierarchical model with the latent context. Instead, they often use mismatched priors to form a linear minimum mean-squared error estimate of the channel parameter, which is then used to estimate successive, unknown transmitted symbols. We make the basic connection that transformers show excellent contextual sequence completion with a few prompts, and so they should be able to implicitly determine the latent context from pilot symbols to perform end-to-end in-context estimation of transmitted symbols. Furthermore, the transformer should use information efficiently, i.e., it should utilize any pilots received to attain the best possible symbol estimates. Through extensive simulations, we show that in-context estimation not only significantly outperforms standard approaches, but also achieves the same performance as an estimator with perfect knowledge of the latent context within a few context examples. Thus, we make a strong case that transformers are efficient in-context estimators in the communication setting. Recent advances in our understanding of transformers have brought to the fore the notion that they are capable of in-context learning. The transformer itself is pre-trained, either implicitly or explicitly over a variety of contexts and so acquires the ability to generate in-distribution outputs conditioned on a specific context.