Deep Learning Practice and Theory

Local representation vs distributed representation l Local representation each concept is represented by one symbol e.g. Giraff 1, Panda 2, Lion 3, Tiger 4 no interfere, noise immunity, precise l Distributed representation each concept is represented by a set of symbol, and each symbol participates in representing many concepts Generalizable less accurate interfere Giraff Pand Lion Tiger Long neck four legs body hair paw pad 61. High dimensional vector vs low dimensional data l High dimensional vector Random two vectors are always almost orthogonal many concepts can be stored within one vector u w x y z, Same characteristics as local representation l Low dimensional vector Interfere each other Cannot keep precise memory Beneficial for generalization l Interference and generalization are strongly related 62. Two layer feedforward network memory augmented network [Vaswani 17] l Memory augmented network a V Softmax(Kq) K is a key matrix (i-th row corresponds to a key for i-th memory) V is a value matix.