It is commonly believed that scaling language models should commit a significant space or time cost, by increasing the parameters (parameter scaling) or output tokens (inference-time scaling). We introduce another and more inference-efficient scaling paradigm: increasing the model's parallel computation during both training and inference time. We apply P diverse and learnable transformations to the input, execute forward passes of the model in parallel, and dynamically aggregate the P outputs. This method, namely parallel scaling (PARSCALE), scales parallel computation by reusing existing parameters and can be applied to any model structure, optimization procedure, data, or task. We theoretically propose a new scaling law and validate it through large-scale pre-training, which shows that a model with P parallel streams is similar to scaling the parameters by O(logP) while showing superior inference efficiency. For example, PARSCALE can use up to 22 less memory increase and 6 less latency increase compared to parameter scaling that achieves the same performance improvement. It can also recycle an off-the-shelf pre-trained model into a parallelly scaled one by post-training on a small amount of tokens, further reducing the training budget. The new scaling law we discovered potentially facilitates the deployment of more powerful models in low-resource scenarios, and provides an alternative perspective for the role of computation in machine learning. Our code and 67 trained model checkpoints are publicly available at https://github.com/QwenLM/ParScale

It is commonly believed that scaling language models should commit a significant space or time cost, by increasing the parameters (parameter scaling) or output tokens (inference-time scaling). We introduce another and more inference-efficient scaling paradigm: increasing the model's parallel computation during both training and inference time. We apply $P$ diverse and learnable transformations to the input, execute forward passes of the model in parallel, and dynamically aggregate the $P$ outputs. This method, namely parallel scaling (ParScale), scales parallel computation by reusing existing parameters and can be applied to any model structure, optimization procedure, data, or task. We theoretically propose a new scaling law and validate it through large-scale pre-training, which shows that a model with $P$ parallel streams is similar to scaling the parameters by $\mathcal O(\log P)$ while showing superior inference efficiency. For example, ParScale can use up to 22$\times$ less memory increase and 6$\times$ less latency increase compared to parameter scaling that achieves the same performance improvement. It can also recycle an off-the-shelf pre-trained model into a parallelly scaled one by post-training on a small amount of tokens, further reducing the training budget. The new scaling law we discovered potentially facilitates the deployment of more powerful models in low-resource scenarios, and provides an alternative perspective for the role of computation in machine learning. Our code and 67 trained model checkpoints are publicly available at https://github.com/QwenLM/ParScale

On the evening of January 6, 2021, Brad Parscale texted Donald Trump adviser Katrina Pierson about the insurrectionist assault on the US Capitol that had finally been quashed by police. "This is about [T]rump pushing for uncertainty in our country," wrote Parscale, who ran digital and data operations for Trump's 2016 campaign and managed his 2020 reelection effort before being replaced. This week I feel guilty for helping him win." "You did what you felt right at the time and therefore it was right," Pierson replied. "Yeah," Parscale answered, "but a woman is dead." The conversation continued, with Pierson texting, "You do realize this was going to happen." Parscale responded that Trump's rhetoric had "killed someone." Pierson countered, "It wasn't the rhetoric." Parscale was obviously blaming Trump for the storming of the Capitol and the death of Trump supporter Ashli Babbitt. In these private texts--which were not made public until mid-2022 during the House investigation of January ...

Former Trump campaign manager reacts to 2020 election results in exclusive interview on'The Story' Former Trump 2020 campaign manager Brad Parscale has accused "D-level" "talking heads" in the president's orbit of starting a whisper campaign that forced him out earlier this year. Speaking with Fox News' Martha MacCallum in an exclusive interview on "The Story" Tuesday night, Parscale alleged that "when the polling numbers were going down, they were in his ear and I was out working." Discussing a reported incident in which Trump berated Parscale for passing along a bad polling report, the former campaign manager said: "I didn't like lying to him -- I like telling the truth. Sometimes that comes with a lot of painful days, knowing that I might let him down or make him upset, but a lot of the D-level people that hung around him told him what he wanted to hear: They were'yes' men. I wasn't going to be'yes' man, but a'get it done' man." Parscale did not name anyone as being specifically responsible for his ouster in mid-July, when he was replaced by Bill Stepien.