Learning to represent tasks for few-shot learning (Communication, Part 1) ARAYA Inc.

This is the first in a multi-part series of posts about our experiments with making neural networks that learn not only via gradient descent, but also by the accumulation of information in their hidden states. The eventual goal that this move towards is to train networks to share information with each-other, so that a population of neural networks could continue to learn and improve even if the weight updates were shut off or the individual networks were swapped out. This touches on a number of distinct topics along the way, so rather than make one big confused mess, I'm going to try to separate them into individual, modular posts. The first step in all of this is to think about how to represent the overall problem I'm asking the networks to solve. The usual situation is that I have some fixed data set or context, a fixed goal given that data set, and then I chop it up into statistically stationary and uniform minibatches of individual examples.