An image showing how machines learn from vision, language, and sound together. Most of us have watched television with the sound turned off at one time or another. While it's usually possible to follow the story at least to some degree, the absence of an audio track tends to limit our ability to fully appreciate what's taking place. Similarly, it's easy to miss a lot of information just listening to the sounds coming from another room. The multimodality of combining image, sound and other details greatly enhances our understanding of what's happening, whether it's on TV or in the real world.

Most of us have watched television with the sound turned off at one time or another. While it's usually possible to follow the story at least to some degree, the absence of an audio track tends to limit our ability to fully appreciate what's taking place. Similarly, it's easy to miss a lot of information just listening to the sounds coming from another room. The multimodality of combining image, sound and other details greatly enhances our understanding of what's happening, whether it's on TV or in the real world. The same appears to be true for artificial intelligence.