We introduce Virtual Width Networks (VWN), a framework that delivers the benefits of wider representations without incurring the quadratic cost of increasing the hidden size. VWN decouples representational width from backbone width, expanding the embedding space while keeping backbone compute nearly constant. In our large-scale experiment, an 8-times expansion accelerates optimization by over 2 times for next-token and 3 times for next-2-token prediction. The advantage amplifies over training as both the loss gap grows and the convergence-speedup ratio increases, showing that VWN is not only token-efficient but also increasingly effective with scale. Moreover, we identify an approximately log-linear scaling relation between virtual width and loss reduction, offering an initial empirical basis and motivation for exploring virtual-width scaling as a new dimension of large-model efficiency.

Only the constraints are presented here. Then, eq. 2 can be reformulated as follow: The complete optimal allocation of eq. 3 can be summarized by the following python script: """EF evaluation """ import copy import logging import os import cvxopt import numpy as np scalar = 10000 def cvxopt_solve_qp(P, q, G= None, h= None, **kwargs): P = 0.5 * (P + P.T) # make sure P is symmetric args = [cvxopt.matrix(P), The remaining two cases are additional edge cases related to the previous condition. The size and description of the dataset we used are presented in table. (see Table 6).

For a detailed treatment, please refer to [1]. As mentioned in Section 3.1 of the main text, in its simplest form, self-attention is described as: y = σ ( QK We have highlighted the same terms with the same color in Equations 2 and 3 to show the results are indeed identical. As discussed in Section 3.2 of the main text, this formulation lets us convert an observation signal Table 1 in the main text contains the hyper-parameters used for each experiment. Applying softmax to each row only brings scalar multipliers to each row and the proof still holds. Computing Engines (GCE) on an instance that has one V100 GPU.

Autoformer achieves sharp improvement over the state-of-the-art on various forecasting horizons. These results show a 60% average MSE reduction over previous state-of-the-art. We fix the input length of Autoformer as 96. For the ILI dataset without obvious periodicity, the larger factor may bring noises. We fix the forecasting horizon as 48 for ILI and 336 for the others.

Guo. Swin transformer: Hierarchical vision transformer using shifted windows.

We use SGD with the momentum of 0.9 and set the batch size as 64 across 8 V100 GPU We use dropout of 0.3 before the final We use dropout of 0.5 before the final classifier. For FineGym [8], we sample a single clip consists of 8 frames for inference. All the benchmarks that we used are commonly used datasets for the academic purpose. As described in Sec.4.2, we For ease description, the notation of multi-query L is omitted. In Figure 2, We provide pseudo-codes of Eq.11 and 12 in Sec.4.2 in our For ease description, the notation of multi-query L is omitted.

In this section, we introduce more detailed background knowledge.