Video SemNet: Memory-Augmented Video Semantic Network