Understanding object motions and transformations is a core problem in computer science. Modeling sequences of evolving images may provide better representations and models of motion and may ultimately be used for forecasting or simulation. Diagrammatic Abstract Reasoning is an avenue in which diagrams evolve in complex patterns and one needs to infer the underlying pattern sequence and generate the next image in the sequence. For this, we develop a novel Contextual Generative Adversarial Network based on Recurrent Neural Networks (Context-RNN-GANs), where both the generator and the discriminator modules are based on contextual history and the adversarial discriminator guides the generator to produce realistic images for the particular time step in the image sequence. We employ the Context-RNN-GAN model (and its variants) on a novel dataset of Diagrammatic Abstract Reasoning as well as perform initial evaluations on a next-frame prediction task of videos. Empirically, we show that our Context-RNN-GAN model performs competitively with 10th-grade human performance but there is still scope for interesting improvements as compared to college-grade human performance.

We propose a learning agent that first learns concepts in an integrated, cross-modal manner, and then uses these as the semantics model to map language. We consider an abstract model for the action of throwing, modeling the entire trajectory. From a large set of throws, we take the trajectory images and and the throwing parameters. These are mapped jointly onto a low-dimensional non-linear manifold. Such models improve with practice, and can be used as the starting point for real-life tasks such as aiming darts or recognizing throws by others. How can such models can be used in learning language? We consider a set of videos involving throwing and rolling actions. These actions are analyzed into a set of contrastive semantic classes based on agent, action, and the thrown object (trajector). We obtain crowdsourced commentaries for these videos (raw text) from a number of adults. The learner attempts to associate labels using contrastive probabilities for the semantic class. Only a handful of high-confidence words are found, but the agent starts off with this partial knowledge. These are used to learn incrementally larger syntactic patterns, initially for the trajector, and eventually for full agent-trajector-action sentences. We demonstrate how this may work for two completely different languages - English and Hindi, and also show how rudiments of agreement, synonymy and polysemy are detected.

Metaphors being at the heart of our language and thought process, computationally modelling them is imperative for reproducing human cognitive abilities. In this work, we propose a plausible grounded cognitive model for artificial metaphor acquisition. We put forward a rule-based metaphor acquisition system, which doesn't make use of any prior 'seed metaphor set'. Through correlation between a video and co-occurring commentaries, we show that these rules can be automatically acquired by an early learner capable of manipulating multi-modal sensory input. From these grounded linguistic concepts, we derive classes based on lexico-syntactical language properties. Based on the selectional preferences of these linguistic elements, metaphorical mappings between source and target domains are acquired.

Human wayfinding operates very very differently from traditional deterministic algorithms owing to a) restrictions in working memory resulting in subjective regionalized maps, and b)flexible adoption of different navigation strategies. While a number of cognitive strategies have been proposed for human wayfinding, these have been hard to evaluate thoroughly owing to a lack of computational simulation. In this work, we propose a stochastic approach for capturing these aspects, and argue for a memoryless, stationary implementation. In two longitudinal experiments on the same group of subjects, we first estimate the subjective regionalized maps for each subject on the same familiar spatial domain. Later, based on their wayfinding responses, we can estimate the stationary probabilities for different strategies. We apply this algorithm to evaluate three wayfinding strategies proposed in the literature, and repudiate the previously held suggestion that they are followed equiprobably.