Learning Joint Statistical Models for Audio-Visual Fusion and Segregation

People can understand complex auditory and visual information, often using one to disambiguate the other. Automated analysis, even at a lowlevel, facessevere challenges, including the lack of accurate statistical models for the signals, and their high-dimensionality and varied sampling rates.Previous approaches [6] assumed simple parametric models for the joint distribution which, while tractable, cannot capture the complex signalrelationships. We learn the joint distribution of the visual and auditory signals using a nonparametric approach. First, we project the data into a maximally informative, low-dimensional subspace, suitable for density estimation. These learned densities allow processing across signal modalities.