High-quality pretraining data is the fossil fuel of large language models (LLMs), yet its reserves are running low for frontier models. In this paper, we introduce RePro, a novel web recycling method that trains a relatively small LM with reinforcement learning to generate effective and faithful rephrasings of pretraining data. Specifically, we design one quality reward and three faithfulness rewards, optimizing the LM rephraser to convert organic data into high-quality rephrasings while maintaining its core semantics and structure. In our experiment, we train a 4B rephraser to recycle 72B tokens sampled from DCLM-RefinedWeb. Pretraining results on 400M and 1.4B models demonstrate that RePro delivers 4.7%-14.0% relative accuracy gains over organic-only baseline on 22 downstream tasks. RePro also outperforms ReWire, the state-of-the-art web recycling method that prompts a 70B rephraser, as well as the organic baseline with a 4x larger data pool. Experiments with different amounts of recycled data highlight that RePro improves organic data efficiency by 2-3x. Individual and distributional analyses validate that RePro preserves more critical information and faithfully reflects the characteristics of organic data compared to prompting-based methods. Together, these results show that RePro provides an efficient and controllable path to effectively harness the fossil fuel of LLM pretraining. We open-source our code, rephraser, and recycled data at https://github.com/cxcscmu/RePro.

From the hunt for the philosopher's stone to the snake oil salesmen of the Wild West, the history of medicine has had more than its fair share of fraudulent'cure-alls'. So, when I first heard of a device that claimed to cure everything from depression to my rapidly deteriorating attention span, I was understandably sceptical. To make things even stranger, this potential wonder-cure isn't a pill, powder, or trendy new supplement. Instead, the Nurosym is a 599 gadget that claims to rewire your nervous system - by zapping your ear. MailOnline's Wiliam Hunter bravely tested it out - and, as strange as it all might sound, he's almost ready to believe the hype.

The human brain is capable of integrating intricate and diverse inputs from multiple sensory systems simultaneously in order to rapidly comprehend and assess the information for planning and execution of complex actions. These and other cognitive functions are performed by the vastly interconnected neural networks formed by the roughly 100 billion neurons of the brain. The precise patterns of connectivity among neurons within these networks determines their function, and through experience, these connectivity patterns change over time to enable acquisition of new skills. Indeed, one of the most impressive aspects of brain function is the ability to learn new cognitive skills, such as the ability to understand and speak a foreign language. During learning, connections between neurons change through a process known as synaptic plasticity, which plays a pivotal role in learning, While traditional learning brings about changes in neural networks through experience, synaptic plasticity can also be enhanced by activating neuromodulatory regions in the brain via peripheral neurostimulation.

Recommender systems (RS) mediate human experience online. Most RS act to optimize metrics that are imperfectly aligned with the best-interest of users but are easy to measure, like ad-clicks and user engagement. This has resulted in a host of hard-to-measure side-effects: political polarization, addiction, fake news. RS design faces a recommender alignment problem: that of aligning recommendations with the goals of users, system designers, and society as a whole. But how do we test and compare potential solutions to align RS? Their massive scale makes them costly and risky to test in deployment. We synthesized a simple abstract modelling framework to guide future work. To illustrate it, we construct a toy experiment where we ask: "How can we evaluate the consequences of using user retention as a reward function?" To answer the question, we learn recommender policies that optimize reward functions by controlling graph dynamics on a toy environment. Based on the effects that trained recommenders have on their environment, we conclude that engagement maximizers generally lead to worse outcomes than aligned recommenders but not always. After learning, we examine competition between RS as a potential solution to RS alignment. We find that it generally makes our toy-society better-off than it would be under the absence of recommendation or engagement maximizers. In this work, we aimed for a broad scope, touching superficially on many different points to shed light on how an end-to-end study of reward functions for recommender systems might be done. Recommender alignment is a pressing and important problem. Attempted solutions are sure to have far-reaching impacts. Here, we take a first step in developing methods to evaluating and comparing solutions with respect to their impacts on society.

One of the reasons for the brain's incredible power is its ability to rewire itself as it learns. Now researchers have created electronic circuits that can do the same. Efforts to mimic the brain in silicon--a field known as neuromorphic computing--have a long pedigree, and have seen significant investments from computing powerhouses like Intel and IBM. So far, most research has focused on replicating the functionality and connectivity of biological neurons and synapses in the hope of replicating the brain's incredible learning efficiency. One feature of neurons that has received less attention is the way they're able to reorganize themselves in response to experience.

Despite the promise of artificial intelligence (AI) and machine learning (ML), the greatest challenge of all is not data or process or technology -- it is culture. Such technologies have promised automation galore and insight discovery at the flip of a switch. But culture continues to impede our ability to realize this promise. Data and analytics leaders need to put critical touchstones in place today to ensure the right decisions are taken in the right way at the right time and for the right outcome. This special report provides the insight, best practices and critical touchstones you need to set up now in order to help rewire your culture.

The ticket kiosks at Shanghai's frenetic subway station have a mind of their own. Walk up to one and state your destination, and it'll automatically recommend a route before issuing a ticket. In the interest of reducing the rush-hour stampede, the system is set up to let you find information and buy tickets without pushing a button or talking to a person. More impressive still, all this happens successfully in the middle of a crowded, noisy station. Each kiosk has to figure out who is speaking to it; zero in on that person's voice within the crowd; transcribe the incoming speech; parse its meaning; and compare the person's face against a massive database of photos--all within a few seconds.

The saying goes that'power corrupts', and a new study suggests there may be some truth behind this - especially when it comes to brain function. Researchers have found that CEOs and other leaders may suffer damage to their brain as a result of their rise to power. The damage results in the loss of the ability to read other people's emotions, which could explain why people who achieve great power lose their ability to feel empathy for the less powerful. Neuroscientists at McMaster University in Ontario used transcranial magnetic stimulation to discover the lack of empathy in people who feel powerful. Forty-five volunteers took part in the study.

Learning to read and wires rewires the brain in just six months, according to new research. A study of women in India who learned to read in their 30s has shown the human brain's incredible capacity to reorganise and transform itself, researchers said. Researchers recruited women in India, a country with an illiteracy rate of around 39 per cent, to see what they could learn about the areas of the brain devoted to reading. Learning to read and wires rewires the brain in just six months, according to new research. Pictured is an artist's impression of the brain Researchers studying Indian women learning to read for the first time in their 30s found looking at books can prompt the brain to reorganise itself.