A new immersive classroom uses AI and VR to teach Mandarin Chinese

In addition to surrounding the students with digital projections of a scene, the environment uses several types of sensors to dynamically adapt to the students' words and actions. Microphones, worn by the participants, feed their audio directly into speech-recognition algorithms. Cameras track their movements and gestures to register when they point to various objects or walk up to different virtual agents. If a student points to a food dish in the restaurant scene and asks what it is, for example, a virtual agent can respond with the name and description. Narrative-generation technology also allows each agent to construct more sophisticated answers to off-the-cuff questions ("What's the dish's history?") using knowledge from Wikipedia.