Can a neural network learn a 2D right-angle of pixels without supervision? • /r/MachineLearning

I'm looking for insight or reading material on a question I don't have a satisfying answer for, although I might just be creating red herrings for myself. Do neural network models learn or memorize? What is the extent to which they can learn feature-functions underlying data, without somebody's previous preconceptions of that relation? There are relatively simple functions that represent this relation, invariant to rotation and translation. To my knowledge, a convolutional neural network requires training data for all three distinct cases - i.e., it would learn these as three separate patterns or features.