Deep Learning Architecture Search and the Adjacent Possible

What is the biggest problem with deep learning networks? I propose that it is: "learning how to forget". Deep learning networks are very good at remembering things, it is so good this capability that it has no trouble learning random labels. Its biggest problem however is that it does not know how to forget. This may surprise many, but learning to forget is critically important in a cognitive system.