A computer reads a story - Visage Technologies

Sequencing is a task for children that aims to improve understanding of the temporal occurrence in a sequence of events, so that they sort various images (sometimes with captions) into a coherent story. Researchers from Virginia Tech and TTI Chicago have proposed the task of machine-learning sequencing – given a jumbled set of aligned image-caption pairs that belong to a story, the task is that the computer needs to sort them in a way that they form a consistent story. They have used stories from the Sequential Image Narrative Dataset where a set of 5 aligned image-caption pairs together form a coherent story, and given a jumbled input story, they have trained machine-learning models to sort them. They have proposed a task of visual story sequencing and implemented two approaches to solve the task. The first one is based on individual story elements to predict the position, and the other one is based on the pairwise story elements to predict the relative order of the story elements.