We consider the problem of training machine learning models on distributed data in a decentralized way. For finite-sum problems, fast single-machine algorithms for large datasets rely on stochastic updates combined with variance reduction. Yet, existing decentralized stochastic algorithms either do not obtain the full speedup allowed by stochastic updates, or require oracles that are more expensive than regular gradients. In this work, we introduce a Decentralized stochastic algorithm with Variance Reduction called DVR. DVR only requires computing stochastic gradients of the local functions, and is computationally as fast as a standard stochastic variance-reduced algorithms run on a $1/n$ fraction of the dataset, where $n$ is the number of nodes. To derive DVR, we use Bregman coordinate descent on a well-chosen dual problem, and obtain a dual-free algorithm using a specific Bregman divergence. We give an accelerated version of DVR based on the Catalyst framework, and illustrate its effectiveness with simulations on real data.

We would like to thank the reviewers for their positive and helpful feedback. Typo: The result of Theorem 1 is on the expected norm of θ indeed, thank you for pointing this out. Derivations can be found in Appendix A. Y et, and although it leads to a related algorithm, our approach is different. We will make sure to insist on the points discussed in this rebuttal in a revised version of the paper. Fastest rates for stochastic mirror descent methods.

We consider the problem of training machine learning models on distributed data in a decentralized way. For finite-sum problems, fast single-machine algorithms for large datasets rely on stochastic updates combined with variance reduction. Yet, existing decentralized stochastic algorithms either do not obtain the full speedup allowed by stochastic updates, or require oracles that are more expensive than regular gradients. In this work, we introduce a Decentralized stochastic algorithm with Variance Reduction called DVR. DVR only requires computing stochastic gradients of the local functions, and is computationally as fast as a standard stochastic variance-reduced algorithms run on a 1/n fraction of the dataset, where n is the number of nodes. To derive DVR, we use Bregman coordinate descent on a well-chosen dual problem, and obtain a dual-free algorithm using a specific Bregman divergence.

We consider the problem of training machine learning models on distributed data in a decentralized way. For finite-sum problems, fast single-machine algorithms for large datasets rely on stochastic updates combined with variance reduction. Yet, existing decentralized stochastic algorithms either do not obtain the full speedup allowed by stochastic updates, or require oracles that are more expensive than regular gradients. In this work, we introduce a Decentralized stochastic algorithm with Variance Reduction called DVR. DVR only requires computing stochastic gradients of the local functions, and is computationally as fast as a standard stochastic variance-reduced algorithms run on a 1/n fraction of the dataset, where n is the number of nodes. To derive DVR, we use Bregman coordinate descent on a well-chosen dual problem, and obtain a dual-free algorithm using a specific Bregman divergence.

Self-supervised learning is well known for its remarkable performance in representation learning and various downstream computer vision tasks. Recently, Positive-pair-Only Contrastive Learning (POCL) has achieved reliable performance without the need to construct positive-negative training sets. It reduces memory requirements by lessening the dependency on the batch size. The POCL method typically uses a single loss function to extract the distortion invariant representation (DIR) which describes the proximity of positive-pair representations affected by different distortions. This loss function implicitly enables the model to filter out or ignore the distortion variant representation (DVR) affected by different distortions. However, existing POCL methods do not explicitly enforce the disentanglement and exploitation of the actually valuable DVR. In addition, these POCL methods have been observed to be sensitive to augmentation strategies. To address these limitations, we propose a novel POCL framework named Distortion-Disentangled Contrastive Learning (DDCL) and a Distortion-Disentangled Loss (DDL). Our approach is the first to explicitly disentangle and exploit the DVR inside the model and feature stream to improve the overall representation utilization efficiency, robustness and representation ability. Experiments carried out demonstrate the superiority of our framework to Barlow Twins and Simsiam in terms of convergence, representation quality, and robustness on several benchmark datasets.

The neocortex stands out as a stunning achievement of biological evolution. All mammals have this swath of tissue covering their brain, and the six layers of densely packed neurons within it handle the sophisticated computations and associations that produce cognitive prowess. Since no animals other than mammals have a neocortex, scientists have wondered how such a complex brain region evolved. Original story reprinted with permission from Quanta Magazine, an editorially independent publication of the Simons Foundation whose mission is to enhance public understanding of science by covering research developments and trends in mathematics and the physical and life sciences. The brains of reptiles seemed to offer a clue.

Birds have complex motor and cognitive abilities that rival or exceed the performance of many mammals, but their brains are organized in a notably different way. Parts of the bird brain have been functionally compared to the mammalian neocortex. However, it is still controversial to what extent these regions are truly homologous with the neocortex or if instead they are examples of evolutionary convergence. Colquitt et al. used single-cell sequencing to identify and characterize the major classes of neurons that comprise the song-control system in birds (see the Perspective by Tosches). They found multiple previously unknown neural classes in the bird telencephalon and shed new light on the long-standing controversy regarding the nature of homology between avian and mammalian brains. Science , this issue p. [eabd9704][1]; see also p. [676][2] ### INTRODUCTION The mammalian neocortex, with its distinctive six-layered structure, is thought to enable advanced cognitive functions not seen in other animals. Yet birds, which have a remarkably different brain organization, display a range of complex motor and cognitive abilities, such as tool use and problem-solving, that are comparable to those of many mammals. Although portions of the avian brain are often compared to the neocortex, especially regions involved in the learning and production of vocalizations, it has remained unclear whether these regions are truly homologous with the neocortex (that is, whether they share a common evolutionary origin) or instead are examples of evolutionary convergence. ### RATIONALE The nature of the similarities and differences in brain organization between mammals and birds has implications for the evolutionary mechanisms that underlie the emergence of advanced behaviors. In mammals, the six-layered neocortex occupies most of the pallium (the outermost portion of the brain), whereas in birds, most of the pallium consists of a distinct unlayered structure called the dorsal ventricular ridge (DVR). The DVR contains multiple interconnected groups of neurons, often referred to as nuclei, that are necessary for complex avian behaviors, including vocal learning in songbirds. Two prevailing viewpoints offer opposing interpretations for how both the mammalian neocortex and avian DVR enable complex behaviors despite their structural differences. One view proposes that the DVR is homologous to the neocortex and that the nuclei in the DVR correspond to distinct layers of the neocortex and thus represent a rearrangement of a conserved ancestral circuit. A second hypothesis argues that the neocortex and DVR develop from distinct embryonic regions of the pallium (dorsal and ventral, respectively) and are therefore nonhomologous structures that separately evolved to serve similar functions. To test these models, we used single-cell transcriptomics to characterize the cell types and gene expression patterns of two regions in the songbird DVR that are necessary for learning and producing birdsong: HVC (proper name) and RA (robust nucleus of the arcopallium). For each type, we characterized the expression profiles of transcription factors, which reflect the cellular identities and regional origins of neurons, and effector genes, which specify neuronal cell properties and function. We compared these profiles with those of neurons previously described in mammals and reptiles to clarify how individual neuronal types are related across amniotes. ### RESULTS We identify a variety of excitatory cell classes that are different between HVC and RA, and inhibitory classes that are shared across regions, similar to organizational patterns of cell types in mammals and reptiles. We show that excitatory neurons in both HVC and RA have transcription factor profiles that bear strong similarity to the mammalian ventral pallium, which includes the olfactory bulb, piriform cortex, and pallial amygdala, but not to the neocortex, which develops from the dorsal pallium. However, when examining only effector genes, we find that excitatory neurons exhibit greater similarity to neocortical neurons from multiple layers and less similarity to the ventral pallium. We also find that songbird inhibitory neurons bear a considerable resemblance to neuron classes in both mammals and turtles, indicating that the major classes of inhibitory neurons are conserved and likely present in ancestral amniotes. We report that, consistent with the interpretation that song-control regions have ventral pallial origins, the most abundant inhibitory neuron type in the songbird DVR is similar to inhibitory neurons that are enriched in mammalian ventral pallial derivatives and absent from the neocortex. ### CONCLUSION Our findings indicate that the avian DVR and the neocortex derive from different neurodevelopmental regions employing distinct transcription factor expression patterns and therefore are not homologous structures. However, we find that excitatory neurons in the DVR have evolved similar properties to the neocortex by engaging overlapping patterns of effector genes. Such overlapping transcriptional profiles may account for the evolution of similar complex motor and cognitive abilities in mammals and birds, including vocal learning, and suggest that the DVR may perform neural computations in a way that is functionally analogous to the neocortex. By addressing a long-standing controversy regarding the relationship between avian and mammalian brains, these results provide insight into the evolution and diversification of neural cell types and structures that enable advanced behaviors. ![Figure][3] Cellular transcriptomics of a songbird vocal circuit. ( A ) Schematic of the song motor pathway (SMP). HVC, proper name; RA, robust nucleus of the arcopallium; DLM, medial portion of the dorsolateral thalamic nucleus; Av, mesopallial auditory nucleus Avalanche; LMAN, lateral magnocellular nucleus of the anterior nidopallium. ( B ) Transcriptional similarities between glutamatergic neurons in the SMP and the mouse neocortex. ( C ) Pallial biases of transcription factor (TF) versus effector gene (non-TF) expression profiles. DP/MP/LP/VP, dorsal/medial/lateral/ventral pallium. ( D ) Diversity and origins of γ-aminobutyric acid–releasing (GABAergic) neurons in the SMP. LGE/MGE/CGE, lateral/medial/caudal ganglionic eminence. Birds display advanced behaviors, including vocal learning and problem-solving, yet lack a layered neocortex, a structure associated with complex behavior in mammals. To determine whether these behavioral similarities result from shared or distinct neural circuits, we used single-cell RNA sequencing to characterize the neuronal repertoire of the songbird song motor pathway. Glutamatergic vocal neurons had considerable transcriptional similarity to neocortical projection neurons; however, they displayed regulatory gene expression patterns more closely related to neurons in the ventral pallium. Moreover, while γ-aminobutyric acid–releasing neurons in this pathway appeared homologous to those in mammals and other amniotes, the most abundant avian class is largely absent in the neocortex. These data suggest that songbird vocal circuits and the mammalian neocortex have distinct developmental origins yet contain transcriptionally similar neurons. [1]: /lookup/doi/10.1126/science.abd9704 [2]: /lookup/doi/10.1126/science.abf9551 [3]: pending:yes

The mere existence of Amazon's Fire TV Recast is a testament to how popular cord-cutting has become. Over-the-air DVR was once the domain of geeks whose living rooms ran on Windows Media Center. But as more people have dropped cable TV, we've seen more user-friendly approaches from the likes of TiVo, Tablo, Plex, Channels, and AirTV. Those options presumably helped inspire the Fire TV Recast, a $230 box that records free broadcast TV channels from an antenna and streams the video to Amazon's popular Fire TV devices. The Fire TV Recast is the most mainstream attempt yet at over-the-air DVR, and it shows in Amazon's simple and polished software.

Cord-cutters are no longer second-class citizens in TiVo's world with the new $250 TiVo Bolt OTA DVR, which is based on the same hardware platform that TiVo offers cable-TV subscribers--less the CableCARD slot. TiVo's previous antenna-only DVR, the Roamio OTA, launched more than four years ago and is based on a cable box from 2013. The new Bolt OTA includes such features 4K UHD video support, voice control, and built-in streaming to smartphones. And while you can technically use the regular Bolt with an antenna, the Bolt OTA's DVR service is less than half the price. Still, the Bolt platform remains outdated in several ways, especially compared to networked DVRs that send video to modern streaming boxes, including Nuvyyo's Tablo, SiliconDust's HDHomeRun, and Amazon's forthcoming Fire TV Recast.