oswaldoludwig/visually-informed-embedding-of-word-VIEW-

The visually informed embedding of word (VIEW) is a continuous vector representation for a word extracted from a deep neural model trained using the Microsoft COCO data set to forecast the spatial arrangements between visual objects, given a textual description. The model is composed of a deep multilayer perceptron (MLP) stacked on the top of a Long Short Term Memory (LSTM) network, the latter being preceded by an embedding layer. The VIEW can be applied to transferring multimodal background knowledge to NLP algorithms, i.e. VIEW can be concatenated to word2vec embedding to improve the encoding of spatial background knowledge. WIEW was evaluated in Spatial Role Labeling (SpRL) algorithms (which recognize spatial relations between objects mentioned in the text) using the Task 3 of SemEval-2013 benchmark data set, SpaceEval.