r/MachineLearning - [R] Peer to Peer Unsupervised Representation Learning

Each node asynchronously trains an unsupervised representation of text. Each trains its own model on its own dataset and learns a representations of language (a projection from raw text to embedding) which their neighbours use as inputs to their own model. As they train, they also validate the representations produced by their neighbours, producing a score using a Fishers information metric. We use distillation to extract knowledge from the peers. The result is a local, transfer capable language model at each node.